Gray Hat Hacking, Second Edition, 2nd Edition

Encoding Shellcode

Some of the many reasons to encode shellcode include

• Avoiding bad characters (\x00, \xa9, etc.)

• Avoiding detection of IDS or other network-based sensors

• Conforming to string filters, for example, tolower

In this section, we will cover encoding of shellcode to include examples.

Simple XOR Encoding

A simple parlor trick of computer science is the “exclusive or” (XOR) function. The XOR function works like this:

0 XOR 0 = 0
0 XOR 1 = 1
1 XOR 0 = 1
1 XOR 1 = 0

The result of the XOR function (as its name implies) is true (Boolean 1) if and only if one of the inputs is true. If both of the inputs are true, then the result is false. The XOR function is interesting because it is reversible, meaning if you XOR a number (bitwise) with ...