In the process of exploitation, vulnerabilities are what everything else builds upon. You can’t have an exploit without an underlying bug. In this case, a bug is an error in the functioning of a program, and a vulnerability is a bug that has security implications. The reliability and robustness of an exploit depends greatly on the qualities of the vulnerability that it takes advantage of. You can’t install a rootkit without first running an exploit. So every aspect of taking over a computer begins with a bug. If software were perfect, security researchers would all be out of a job. Luckily, it isn’t, and Apple’s code is no exception. In this chapter we look at some basic approaches to finding bugs in Leopard. Many of these techniques are general-purpose and would be valid for any piece of software; some are specific to the intricacies of Apple. Since Mac OS X contains both open- and closed-source components, we present approaches for finding vulnerabilities in source code and in binaries for which we don’t have the source code. In addition, we present some clever ways of taking advantage of the open-source public development process used by Apple to identify vulnerabilities in Leopard.
Finding bugs, especially security-critical bugs, is both an art and a science. Some superb bug hunters have difficulty explaining exactly how they find their vulnerabilities; they just follow their gut. Others use a thorough, systematic approach to uncover ...