Learning Malware Analysis

1.3.5 Identifying XOR Encoding

To identify XOR encoding, load the binary in IDA and search for the XOR instruction by selecting Search | text. In the dialog that appears, enter xor and select Find all occurrences as shown here:

When you click on OK, you will be presented with all the occurrences of XOR. It is very common to see the XOR operation where the operands are the same registers, such as xor eax,eax or xor ebx,ebx. These instructions are used by the compiler to zero out register values, and you can ignore these instructions. To identify XOR encoding, look for (a) XOR of a register (or memory reference) with a constant value such as ...