Network Security Tools

How a Buffer Overflows and Why It Matters

A process can allocate memory using stack or heap segments. Heaps allow the allocation of memory dynamically using C functions such as malloc( ), but with this comes the overhead of the OS’s internal dynamic memory allocation routines. Stacks are more convenient for developers because the declaration syntax is simpler, and there is no overhead from dynamic memory allocation routines of the OS.

A stack is a last-in-first-out (LIFO) queue. The common stack operators are push (to add to the end of the stack) and pop (to remove the last item placed on the stack). These operators are used on the Assembly level by instructions with the same name. The stack is 32 bits wide and usually has a static starting position. Its size is governed by the extended base pointer (EBP) and extended stack pointer (ESP) CPU registers, but it typically grows “down.” As it grows, the top of the stack (ESP) gets closer to the lowest virtual memory address, as in Figure 5-2. Also shown in Figure 5-2 is the ESP register, which points to the top of the stack. The EBP register serves a special purpose, as it identifies the start of a stack frame by pointing to the bottom of the current stack frame. A stack frame is an area of memory that holds the local function variables as well as the arguments that were passed to the function that is executing. Stack frames are allocated by subtracting from the value of EBP and moving the bottom of the stack frame up the stack. ...