Practical Malware Analysis

Shellcode Encodings

In order to execute, the shellcode binary must be located somewhere in the program’s address space when it is triggered. When paired with an exploit, this means that the shellcode must be present before the exploit occurs or be passed along with the exploit. For example, if the program is performing some basic filtering on input data, the shellcode must pass this filter, or it will not be in the vulnerable process’s memory space. This means that shellcode often must look like legitimate data in order to be accepted by a vulnerable program.

One example is a program that uses the unsafe string functions strcpy and strcat, both of which do not set a maximum length on the data they write. If a program reads or copies malicious data ...