The Shellcoder's Handbook: Discovering and Exploiting Security Holes, Second Edition

Chapter 14. Protection Mechanisms

With the rise of code execution bugs and exploitation, operating system vendors started adding general protection mechanisms to protect their users. All of these mechanisms (with the exception of code auditing) try to reduce the possibility of a successful exploit but don't make the vulnerability disappear, hence, every minor glitch in the protection mechanisms could be leveraged to re-gain code execution.

This chapter first describes the generalities of the most common mechanisms, and then talks about the specifics of each operating system. It also presents some weaknesses in the protections and what would be needed to bypass them.

Protections

Throughout the chapter there is the need to show different stack layouts as different protections are applied. The following is the C code used as an example:

#include <stdio.h>
#include <string.h>

int function(char *arg) {
   int var1;
   char buf[80];
   int var2;

printf("arg:%p var1:%x var2:%x buf:%x\n", &var1, &var2, buf);
   strcpy(buf, arg);
}

int main(int c, char **v) {
  function(v[1]);
}

The standard stack layout, without any protections or optimizations, looks like the following.

↑ Lower addresses

var2	4 bytes
buf	80 bytes
var1	4 bytes
saved ebp	4 bytes
return address	4 bytes
arg	4 bytes

↓ Higher addresses

Note that the chapter uses the same convention as every known debugger: lower addresses are shown higher up the page.

Non-Executable Stack

A very common class of security bugs, even today, is the stack-based buffer overflow, and ...