The Shellcoder's Handbook: Discovering and Exploiting Security Holes

Chapter 3. Shellcode

Shellcode is defined as a set of instructions injected and then executed by an exploited program. Shellcode is used to directly manipulate registers and the function of a program, so it must be written in hexadecimal opcodes. You cannot inject shellcode written from a high-level language, and there are subtle nuances that will prevent shellcode from executing cleanly. This is what makes writing shellcode somewhat difficult, and also somewhat of a black art. In this chapter, we are going to lift the hood on shellcode and get you started writing your own.

The term shellcode is derived from its original purpose—it was the specific portion of an exploit used to spawn a root shell. This is still the most common type of shellcode used, but many programmers have refined shellcode to do more, which we will cover in this chapter. As you have seen in Chapter 2, shellcode is placed into an input area, and then the program is tricked into executing the supplied shellcode. If you worked the examples in the previous chapter, you have already made use of shellcode that can exploit a program.

Understanding shellcode and eventually writing your own is, for many reasons, an essential hacking skill. First and foremost, in order to determine that a vulnerability is indeed exploitable, you must first exploit it. This may seem like common sense, but quite a number of people out there are willing to state whether a vulnerability is exploitable or not without providing solid evidence. ...