Blue Fox

CHAPTER 3OS Fundamentals

The programs that we want to reverse engineer almost never execute in a vacuum. Instead, programs typically run inside the context of a running operating system, such as Linux, Windows, or macOS. Understanding the fundamentals of how these operating systems expose services, system memory, and hardware isolation to programs is therefore necessary to properly understand how the program will behave when it is eventually run.

OS Architecture Overview

Different operating systems often operate in substantially different ways, but, perhaps surprisingly, the execution environments in which ordinary programs run usually share a lot of similarities. For example, the distinction between kernel mode and user mode, as well as access to memory, scheduling, and system service call mechanisms, tends to be relatively small, even if the underlying implementation and semantics vary slightly from platform to platform.

In this section, we will take a quick look at several of these fundamental operating system concepts. Although the focus will primarily be on Linux here, many of the same basic concepts transfer to other operating systems you might encounter when reverse engineering.

User Mode vs. Kernel Mode

Before reverse engineering a program binary, it is important to understand the context in which programs run within a Linux operating system. Armv8‐A CPUs provide the operating system with at least two execution modes. The privileged mode used by the operating system ...