The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

Chapter 3. Shellcode

Shellcode is defined as a set of instructions injected and then executed by an exploited program. Shellcode is used to directly manipulate registers and the function of a program, so it is generally written in assembler and translated into hexadecimal opcodes. You cannot typically inject shellcode written from a high-level language, and there are subtle nuances that will prevent shellcode from executing cleanly. This is what makes writing shellcode somewhat difficult, and also somewhat of a black art. This chapter lifts the hood on shellcode and gets you started writing your own.

The term shellcode is derived from its original purpose—it was the specific portion of an exploit used to spawn a root shell. This is still the most common type of shellcode used, but many programmers have refined shellcode to do more, which is covered in this chapter. As you saw in Chapter 2, shellcode is placed into an input area, and then the program is tricked into executing the supplied shellcode. If you worked through the examples in the previous chapter, you have already made use of shellcode that can exploit a program.

Understanding shellcode and eventually writing your own is, for many reasons, an essential skill. First and foremost, in order to determine that a vulnerability is indeed exploitable, you must first exploit it. This may seem like common sense, but quite a number of people out there are willing to state whether or not a vulnerability is exploitable without providing ...