Understanding the Linux Kernel, 3rd Edition

Entering and Exiting a System Call

Native applications^[*] can invoke a system call in two different ways:

• By executing the int $0x80 assembly language instruction; in older versions of the Linux kernel, this was the only way to switch from User Mode to Kernel Mode.

• By executing the sysenter assembly language instruction, introduced in the Intel Pentium II microprocessors; this instruction is now supported by the Linux 2.6 kernel.

Similarly, the kernel can exit from a system call—thus switching the CPU back to User Mode—in two ways:

• By executing the iret assembly language instruction.

• By executing the sysexit assembly language instruction, which was introduced in the Intel Pentium II microprocessors together with the sysenter instruction.

However, supporting two different ways to enter the kernel is not as simple as it might look, because:

• The kernel must support both older libraries that only use the int $0x80 instruction and more recent ones that also use the sysenter instruction.

• A standard library that makes use of the sysenter instruction must be able to cope with older kernels that support only the int $0x80 instruction.

• The kernel and the standard library must be able to run both on older processors that do not include the sysenter instruction and on more recent ones that include it.

We will see in the section "Issuing a System Call via the sysenter Instruction" later in this chapter how the Linux kernel solves these compatibility problems.

Issuing a System Call via the int $0x80 ...