The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

Chapter 5. Introduction to Heap Overflows

This chapter focuses on heap overflows on the Linux platform, which uses a malloc implementation originally written by Doug Lee, hence called dlmalloc. This chapter also introduces concepts that will help you when facing any other malloc() implementation. Indeed, writing a heap overflow is a rite of passage that teaches you how to think beyond grabbing EIP from a saved stack pointer. dlmalloc is just one library out of many that stores important meta-data interspersed with user data. Understanding how to exploit malloc bugs is a key to finding innovative ways to exploit bugs that don't fit into any particular category.

Doug Lee himself has a terrific summary of dlmalloc on his Web site, at http://gee.cs.oswego.edu/dl/html/malloc.html. If you are unfamiliar with the Doug Lee malloc implementation, you should read it before going on with this chapter. Although his text goes over the concepts you'll need to be familiar with during exploitation, various changes have been made in modern glibc to his original implementation to make it multithreaded and optimized for various situations.

What Is a Heap?

When a program is running, each thread has a stack where local variables are stored. But for global variables, or variables too large to fit on the stack, the program needs another section of writable memory available as a storage space. In fact, it may not know at compile time how much memory it will need, so these segments are often allocated at runtime, ...