Chapter 14. Protection Mechanisms

With the rise of code execution bugs and exploitation, operating system vendors started adding general protection mechanisms to protect their users. All of these mechanisms (with the exception of code auditing) try to reduce the possibility of a successful exploit but don't make the vulnerability disappear, hence, every minor glitch in the protection mechanisms could be leveraged to re-gain code execution.

This chapter first describes the generalities of the most common mechanisms, and then talks about the specifics of each operating system. It also presents some weaknesses in the protections and what would be needed to bypass them.


Throughout the chapter there is the need to show different stack layouts as different protections are applied. The following is the C code used as an example:

#include <stdio.h>
#include <string.h>

int function(char *arg) {
   int var1;
   char buf[80];
   int var2;
printf("arg:%p var1:%x var2:%x buf:%x\n", &var1, &var2, buf);
   strcpy(buf, arg);

int main(int c, char **v) {

The standard stack layout, without any protections or optimizations, looks like the following.

↑ Lower addresses


4 bytes


80 bytes


4 bytes

saved ebp

4 bytes

return address

4 bytes


4 bytes

↓ Higher addresses

Note that the chapter uses the same convention as every known debugger: lower addresses are shown higher up the page.

Non-Executable Stack

A very common class of security bugs, even today, is the stack-based buffer overflow, and ...

Get The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition now with O’Reilly online learning.

O’Reilly members experience live online training, plus books, videos, and digital content from 200+ publishers.