Chapter 1. Introduction
Linux was first released into an unsuspecting world in the summer of 1991. Initially the spare-time hobby of a Finnish computer scientist by the name of Linus Torvalds, Linux was at first accessible only in software source code form to those with enough expertise to build and install it. Early enthusiasts (most also developers themselves by necessity) exploited the growth of the Internet in the early 1990s as a means to build online communities and drive development forward. These communities helped to build the first Linux software distributions, containing all the software components needed to install and use a Linux system without requiring users to be technical experts.
Over the next decade, Linux grew into the mature Unix-like operating system it is today. Linux now powers anything and everything from the smallest handheld gadget to the largest supercomputing cluster, and a nearly infinite range of different devices in between. Examples of the wide range of Linux use abound all around: digital TV receivers and recorders such as TiVo, cell phones from big names like Motorola, Hollywood’s huge Linux “render farms” (used to generate many of the recent CGI movies we have seen), and household name websites such as Google. In addition, a growing number of multinational corporations have successfully built businesses selling Linux software.
In many ways, Linux came along at the right moment in time. But it owes a lot of its success to the work of projects that came before it. Without the hard work of Richard Stallman and the Free Software Foundation (FSF) over the decade prior to Linux arriving on the scene, many of the tools needed to actually build and use a Linux system would not exist. The FSF produced the GNU C Compiler (GCC) and many of the other tools and utilities necessary for building your own embedded Linux systems from scratch, or at least from pre-built collections of these tools that are supplied by third-party vendors. Software maintained by the Free Software Foundation comprises a collection known as GNU, for “GNU’s Not UNIX,” also known (to some) as the GNU system. This stemmed from the FSF’s stated goal to produce a free Unix-like system.
Embedded systems running Linux are the focus of this book. In many ways, these are even more ubiquitous than their workstation and server counterparts—mostly due to the sheer volume of devices and consumer gadgets that rely upon Linux for their operation. The embedded space is constantly growing with time. It includes obvious examples, such as cellular telephones, MP3 players, and a host of digital home entertainment devices, but also less-obvious examples, such as bank ATMs, printers, cars, traffic signals, medical equipment, technical diagnostic equipment, and many, many more. Essentially, anything with a microprocessor that is not considered a “computer” but performs some kind of function using computing is a form of embedded system.
If you are reading this book, you probably have a basic idea why one would want to run an embedded system using Linux. Whether because of its flexibility, its robustness, its price tag, the community developing it, or the large number of vendors supporting it, there are many reasons for choosing to build an embedded system with Linux and many ways to carry out the task. This chapter provides the background for the material presented in the rest of the book by discussing definitions, real-life issues, generic embedded Linux systems architecture, and methodology. This chapter sets the stage for later chapters, which will build upon concepts introduced here.
Definitions
The words “Linux,” “embedded Linux,” and “real-time Linux” are often used with little reference to what is actually being designated with such terminology. Sometimes, the designations may mean something very precise, whereas other times, a broad range or a category of application is meant. In this section, you will learn what the use of these terms can mean in a variety of different situations—starting with the many meanings of “Linux.”
What Is Linux?
Technically speaking, Linux refers only to an operating system kernel originally written by Linus Torvalds. The Linux kernel provides a variety of core system facilities required for any system based upon Linux to operate correctly. Application software relies upon specific features of the Linux kernel, such as its handling of hardware devices and its provision of a variety of fundamental abstractions, such as virtual memory, tasks (known to users as processes), sockets, files, and the like. The Linux kernel is typically started by a bootloader or system firmware, but once it is running, it is never shut down (although the device itself might temporarily enter a low-powered suspended state). You will learn more about the Linux kernel in Chapter 5.
These days, the term “Linux” has become somewhat overloaded in everyday communication. In large part, this is due to its growing popularity—people might not know what an operating system kernel is or does, but they will have perhaps heard of the term Linux. In fact, Linux is often used interchangeably in reference to the Linux kernel itself, a Linux system, or an entire prebuilt (or source) software distribution built upon the Linux kernel and related software. Such widely varying usage can lead to difficulties when providing technical explanations. For example, if you were to say, “Linux provides TCP/IP networking,” do you mean the TCP/IP stack implementation in the Linux kernel itself, or the TCP/IP utilities provided by a Linux distribution using the Linux kernel, or all of the above?
The broadness of the usage of the term has led to calls for a greater distinction between uses of the term “Linux.” For example, Richard Stallman and the Free Software Foundation often prefix “GNU/” (as in “GNU/Linux”) in order to refer to a complete system running a Linux kernel and a wide variety of GNU software. But even terms such as these can be misleading—it’s theoretically possible to build a complete Linux-based system without GNU software (albeit with great difficulty), and most practical Linux systems make use of a variety of both GNU and non-GNU software. Despite the confusion, as more people continue to hear of Linux, the trend is toward a generalization of the term as a reference to a complete system or distribution, running both GNU and non-GNU software on a Linux kernel. If a friend mentions that her development team is using Linux, she probably means a complete system, not a kernel.
A Linux system may be custom built, as you’ll see later, or it can be based on an already available distribution. Despite a growth in both the availability of Linux distributions targeted at embedded use, and their use in embedded Linux devices, your friend’s development team may well have custom built their own system from scratch (for reasons explained later in this book). Conversely, when an end user says she runs Linux on the desktop, she most likely means that she installed one of the various distributions, such as Red Hat Enterprise Linux (RHEL), SuSE Linux Enterprise Server (SLES), Ubuntu Linux, or Debian GNU/Linux. The end user’s running Linux system is as much a Linux system as that of your friend’s, but apart from the kernel, their systems most likely have very different purposes, are built from very different software packages, and run very different applications.
When people use the term Linux in everyday conversation, they usually are referring to a Linux distribution, such as those just mentioned. Linux distributions vary in purpose, size, and price, but they share a common goal: to provide the user with a prepackaged, shrinkwrapped set of files and an installation procedure to get the kernel and various overlaid software installed on a certain type of hardware for a certain purpose. In the embedded space, a variety of embedded Linux distributions are available, such as those from MontaVista, Wind River, Timesys, Denx, and other specialist vendors. These specialist embedded Linux distributions are generally not targeted at generic desktop, workstation, or server use like their “mainstream” counterparts. This means that they typically won’t include software that is not suited for embedded use.
Beginning with the next chapter and throughout the remainder of this book, we will frequently avoid referring to the word “Linux” on its own. Instead, we will generally refer directly to the object of discussion, so rather than talking about the “Linux kernel,” the “Linux system,” and the “Linux distribution,” we will generally refer only to the “kernel,” the “system,” and the “distribution,” respectively. In each of these circumstances, “Linux” is obviously implied. We will use the term “Linux,” where appropriate, to designate the broad range of software and resources surrounding the kernel.
What Is Embedded Linux?
Embedded Linux typically refers to a complete system, or in the context of an embedded Linux vendor, to a distribution targeted at embedded devices. Although the term “embedded” is often also used in kernel discussions (especially between developers who have “embedded concerns”—words often used in the community), there is no special form of the Linux kernel targeted at embedded applications. Instead, the same Linux kernel source code is intended to be built for the widest range of devices, workstations, and servers imaginable, although obviously it is possible to configure a variety of optional features according to the intended use of the kernel. For example, it is unlikely that your embedded device will feature 128 processors and terrabytes of memory, and so it is possible to configure out support for certain features typically found only on larger Linux systems. Chapter 5 covers the kernel in much greater detail, including where to get source code, embedded concerns, and how to build it yourself.
In the context of embedded development, you will typically encounter embedded Linux systems—devices that use the Linux kernel and a variety of other software—and embedded Linux distributions—a prepackaged set of applications tailored for embedded systems and development tools to build a complete system. It is the latter that you are paying for when you go to an embedded Linux vendor. They provide development tools such as cross-compilers, debuggers, project management software, boot image builders, and so on. A growing number of vendors have chosen to integrate much of this functionality into customized plug-ins for their own versions of the community-developed Eclipse graphical IDE framework, which you will learn more about later in this book.
Whether you use a vendor is entirely up to you—few of the examples mentioned in this book will make any assumption as to your reliance or otherwise on a Linux vendor. In fact, much of this book is intended to equip you to build your own tools and tailored Linux distributions. This helps both those who want to use vendor supplied tools and those who do not. Understanding is key in either case, since greater understanding will help you to get more done faster. The bottom line is, of course, about time and resources. Even though this book will help you, should you wish to go it alone, you may choose to buy into an embedded Linux vendor as a way to reduce your product time to market (and to have someone to yell at if things don’t work out according to plan).
This book exclusively discusses embedded Linux systems, and therefore there is no need to keep repeating “embedded Linux” in every name. In general, we will refer to the host system used for developing the embedded Linux system as the “host system,” or “host” for short. The target, which will be the embedded Linux system, will be referred to as the “target system,” or “target.” Distributions providing development frameworks will be referred to as “development distributions” or something similar. This kind of nomenclature should be familiar to anyone who has experience working with embedded systems.[3] Distributions that provide tailored software packages will be referred to as “target distributions.”
What Is Real-Time Linux?
Initially, “Real-Time Linux” uniquely designated the RTLinux project released in 1996 by Michael Barabanov under Victor Yodaiken’s supervision. The original goal of the project was to provide a mechanism for deterministic response times under a Linux environment. Later, the project was expanded to support much more than the originally intended applications, and today supports a variety of non-embedded uses, such as real-time stock market trading systems and other “enterprise” applications. RTLinux was sold to Wind River in early 2007.
Today, there are several other big name real-time projects for Linux, including one that is aiming to add real-time support to the official Linux kernel. You will learn much more about these projects in the latter chapters of this book (Chapter 12 onward), including coverage of some of the innovative concepts and development ideas being worked on. Of course, by the time you read this book much of this technology may be even more commonplace than it is now, especially once real-time capabilities are available in every kind of Linux system installed from here to Timbuktu.
Real Life and Embedded Linux Systems
What types of embedded systems are built with Linux? Why do people choose Linux? What issues are specific to the use of Linux in embedded systems? How many people actually use Linux in their embedded systems? How do they use it? All these questions and many more come to mind when pondering the use of Linux in an embedded system. Finding satisfactory answers to the fundamental questions is an important part of building the system. This isn’t just a general statement. These answers will help you convince management, assist you in marketing your product, and most of all, enable you to evaluate whether your initial expectations have been met.
Types of Embedded Linux Systems
We could use the traditional segments of embedded systems such as aerospace, automotive systems, consumer electronics, telecom, and so on to outline the types of embedded Linux systems, but this would provide no additional information in regard to the systems being designated, because embedded Linux systems may be similarly structured regardless of the market segment. Rather, let us instead classify embedded systems by the criteria that will provide actual information about the structure of the system: size, time constraints, networkability, and degree of intended user interaction with the final system. The following sections cover each of these issues in more depth.
Size
The size of an embedded Linux system is determined by a number of different factors. First, there is physical size. Some systems can be fairly large, like the ones built out of clusters, whereas others are fairly small, like the Linux wristwatches that have been built in partnership with IBM. The physical size of an embedded system is often an important determination of the hardware capabilities of that system (the size of the physical components inside the finished device) and so secondly comes the size of the components with the machine. These are very significant to embedded Linux developers and include the speed of the CPU, the size of the RAM, and the size of the permanent storage (which might be a hard disk, but is often a flash device—currently either NOR or NAND, according to use).
In terms of size, we will use three broad categories of systems: small, medium, and large. Small systems are characterized by a low-powered CPU with a minimum of 4 MB of ROM (normally NOR or even NAND Flash rather than a real ROM) and between 8 and 16 MB of RAM. This isn’t to say Linux won’t run in smaller memory spaces, but it will take you some effort to do so for very little gain, given the current memory market. If you come from an embedded systems background, you may find that you could do much more using something other than Linux in such a small system, especially if you’re looking at “deeply embedded” options. Remember to factor in the speed at which you could deploy Linux, though. You don’t need to reinvent the wheel, like you might well end up doing for a “deeply embedded” design running without any kind of real operating system underneath.
Medium-size systems are characterized by a medium-powered CPU with 32 MB or more of ROM (almost always NOR flash, or even NAND Flash on some systems able to execute code from block-addressable NAND FLASH memory devices) and 64–128 MB of RAM. Most consumer-oriented devices built with Linux belong to this category, including various PDAs (for example, the Nokia Internet Tablets), MP3 players, entertainment systems, and network appliances. Some of these devices may include secondary storage in the form of NAND Flash (as much as 4 GB NAND Flash parts are available at the time of this writing; much larger size arrays are possible by combining more than one part, and we have seen systems using over 32 GB of NAND, even at the time that we are writing this), removable memory cards, or even conventional hard drives. These types of devices have sufficient horsepower and storage to handle a variety of small tasks, or they can serve a single purpose that requires a lot of resources.
Large systems are characterized by a powerful CPU or collection of CPUs combined with large amounts of RAM and permanent storage. Usually these systems are used in environments that require large amounts of calculations to carry out certain tasks. Large telecom switches and flight simulators are prime examples of such systems, as are government research systems, defense projects, and many other applications that you would be unlikely to read about. Typically, such systems are not bound by costs or resources. Their design requirements are primarily based on functionality, while cost, size, and complexity remain secondary issues.
In case you were wondering, Linux doesn’t run on any processor with a memory architecture below 32 bits (certainly there’s no 8-bit microcontroller support!). This rules out quite a number of processors traditionally used in embedded systems. Fortunately though, with the passage of time, increasing numbers of embedded designs are able to take advantage of Linux as processors become much more powerful (and integrate increasing functionality), RAM and Flash prices fall, and other costs diminish. These days, it often makes less economic sense to deploy a new 8051 microcontroller design where for a small (but not insignificant) additional cost one can have all the power of a full Linux system—especially true when using ucLinux-supported devices. The decreasing cost of System-On-Chip (SoC) parts combining CPU/peripheral functionality into a single device is rapidly changing the cost metrics for designers of new systems. Sure, you don’t need a 32-bit microprocessor in that microwave oven, but if it’s no more expensive to use one, and have a built-in web server that can remotely update itself with new features, why not?
Time constraints
There are two types of time constraints for embedded systems: stringent and mild. Stringent time constraints require that the system react in a predefined time frame; otherwise, ca tastrophic events happen. Take for instance a factory where workers have to handle materials being cut by large equipment. As a safety precaution, optical detectors are placed around the blades to detect the presence of the specially colored gloves used by the workers. When the system is alerted that a worker’s hand is in danger, it must stop the blades immediately. It can’t wait for some disk I/O operation involving reading data in from a Linux swap device (for example, swapping back in the memory storing safety management task code) or for some running task to relinquish the CPU. This system has stringent time requirements; it is a hard real-time system. If it doesn’t respond, somebody might lose an arm. Device failure modes don’t get much more painful than that.
Streaming audio systems and consumer devices such as MP3 players and cell phones would also qualify as having stringent requirements, because any transient lagging in audio is usually perceived as bothersome by the users, and failure to contact a cellular tower within a certain time will result in an active call being dropped. Yet, these latter systems would mostly qualify as having soft real-time requirements, because the failure of the application to perform in a timely fashion all the time isn’t catastrophic, as it would be for a hard real-time system. In other words, although infrequent failures will be tolerated—a call being dropped once in a while is an annoying frustration users already live with—the system should be designed to have stringent time requirements. Soft real-time requirements are often the target of embedded Linux vendors that don’t want the (potential) liability of guaranteeing hard real-time but are confident in the abilities of their product to provide, for example, reliable cell phone base-band GSM call management capabilities.
Mild time constraints vary a lot in requirements, but they generally apply to systems where timely responsiveness isn’t necessarily critical. If an automated teller takes 10 more seconds to complete a transaction, it’s generally not problematic (of course, at some point, the user is going to give up on the system and assume it’s never going to respond). The same is true for a PDA that takes a certain number of seconds to start an application. The extra time may make the system seem slow, but it won’t affect the end result. Nonetheless, it’s important that the system make the user aware that it is, in fact, doing something with this time and hasn’t gone out for lunch. Nothing is more frustrating than not knowing whether a system is still working or has crashed.
Networkability
Networkability defines whether a system can be connected to a network. Nowadays, we can expect everything to be accessible through the network, even the refrigerator, toaster, and coffee machine (indeed, a disturbing number of coffee machines can now download new coffee-making recipes online). This, in turn, places special requirements on the systems being built. One factor pushing people to choose Linux as an embedded OS is its proven networking capabilities. Falling prices and standardization of networking components are accelerating this trend. Most Linux devices have one form or another of network capability, be it wired or wireless in nature. The Nokia N770, N800, and N810 Internet Tablets are great examples of embedded Linux devices, complete with 802.11g wireless networking and much more, while the One Laptop Per Child (OLPC) project uses Linux and builds self-assembling, self-managing WiFi mesh networks using 802.11n on the fly.
Networking issues are discussed in detail in Chapter 10.
User interaction
The degree of user interaction varies greatly from one system to another. Some systems, such as PDAs and the Nokia Internet Tablet devices mentioned earlier, are centered around user interaction, whereas others, such as industrial process control systems, might only have LEDs and buttons for interaction (or perhaps even no apparent I/O of any kind). Some other systems have no user interface whatsoever. For example, certain components of an autopilot system in a modern airplane may take care of controlling the wing ailerons but have no direct interaction with the human pilots (something you probably don’t want to consider next time you’re flying).
Reasons for Choosing Linux
There are a wide range of motivations for choosing Linux over a traditional embedded OS. Many of these are shared by those in the desktop, server, and enterprise spaces, while others are more unique to the use of Linux in embedded devices.
Quality and reliability of code
Quality and reliability are subjective measures of the level of confidence in the code that comprises software such as the kernel and the applications that are provided by distributions. Although an exact definition of “quality code” would be hard to agree upon, there are properties many programmers come to expect from such code:
- Modularity and structure
Each separate functionality should be found in a separate module, and the file layout of the project should reflect this. Within each module, complex functionality is subdivided in an adequate number of independent functions. These (simpler) functions are used in combination to achieve the same complex end result.
- Readability
The code should be readable and (more or less) easy to fix for those who understand its internals.
- Extensibility
Adding features to the code should be fairly straightforward. If structural or logical modifications are needed, they should be easy to identify.
- Configurability
It should be possible to select which features from the code should be part of the final application. This selection should be easy to carry out.
The properties expected from reliable code are the following:
- Predictability
Upon execution, the program’s behavior is supposed to be within a defined framework and should not become erratic. Any internal state machine should be consistent in its function, including its error handling.
- Error recovery
In case a problematic situation occurs, it is expected that the program will take steps to recover cleanly from the problem condition and then alert the proper authorities (perhaps a system administrator or the owner of the device running the software in question) with a meaningful diagnostic message.
- Longevity
The program will run unassisted for long periods of time and will conserve its integrity, regardless of the situations it encounters. The program cannot fail simply because a system logfile became too big (something one of the authors of this book admits to having once learned the hard way).
Most programmers agree that the Linux kernel and other projects used in a Linux system fit this description of quality and reliability. The reason is the open source development model (see upcoming note), which invites many parties to contribute to projects, identify existing problems, debate possible solutions, and fix problems effectively. Poor design choices are made from time to time, but the nature of the development model and the involvement of “many eyeballs” serve to more quickly identify and correct such mistakes.
These days you can reasonably expect to run Linux for years unattended without problems, and people have effectively done so. You can also select which system components you want to install and which you would like to avoid. With the kernel, too, you can select which features you would like during build configuration. As a testament to the quality of the code that makes up the various Linux components, you can follow the various mailing lists and see how quickly problems are pointed out by the individuals maintaining the various components of the software or how quickly features are added. Few other OSes provide this level of quality and reliability.
Tip
Strictly speaking, there is no such thing as the “Open Source” development model, or even “Free Software” development model. “Open source” and “Free Software” correspond to a set of licenses under which various software packages can be distributed. Nevertheless, it remains that software packages distributed under “Open Source” and “Free Software” licenses very often follow a similar development model. This development model has been explained by Eric Raymond in his seminal book, The Cathedral and the Bazaar (O’Reilly).
Availability of code
Code availability relates to the fact that Linux’s source code and all build tools are available without any access restrictions. The most important Linux components, including the kernel itself, are distributed under the GNU General Public License (GPL). Access to these components’ source code is therefore compulsory (at least to those users who have purchased any system running GPL-based software, and they have the right to redistribute once they obtain the source in any case). Other components are distributed under similar licenses. Some of these licenses, such as the BSD license, for instance, permit redistribution of binaries without the original source code or the redistribution of binaries based on modified sources without requiring publication of the modifications. Nonetheless, the code for the majority of projects that contribute to the makeup of Linux is readily available without restriction.
When source access problems arise, the open source and free software community seeks to replace the “faulty” software with an open source version that provides similar capabilities. This contrasts with traditional embedded OSes, where the source code isn’t available or must be purchased for very large sums of money. The advantages of having the code available are the possibility of fixing the code without exterior help and the capability of digging into the code to understand its operation. Fixes for security weaknesses and performance bottlenecks, for example, are often very quickly available once the problem has been publicized. With traditional embedded OSes, you have to contact the vendor, alert it of the problem, and await a fix. Most of the time, people simply find workarounds instead of waiting for fixes. For sufficiently large projects, managers even resort to purchasing access to the code to alleviate outside dependencies. Again, this lack of dependence upon any one external entity adds to the value of Linux.
Code availability has implications for standardization and commoditization of components, too. Since it is possible to build Linux systems based entirely upon software for which source is available, there is a lot to be gained from adopting standardized embedded software platforms. As an example, consider the growing numbers of cell phone manufacturers who are working together on common reference software platforms, to avoid re-inventing the same for each new project that comes along (bear in mind that the cell phone market is incredibly volatile, and that a single design might last a year or two if it’s very, very popular). The OpenMoko project is one such effort: a standard Linux-based cell phone platform that allows vendors to concentrate on their other value-adds rather than on the base platform.
Hardware support
Broad hardware support means that Linux supports different types of hardware platforms and devices. Although a number of vendors still do not provide Linux drivers, considerable progress has been made and more is expected. Because a large number of drivers are maintained by the Linux community itself, you can confidently use hardware components without fear that the vendor may one day discontinue driver support for that product line. Broad hardware support also means that, at the time of this writing, Linux runs on dozens of different hardware architectures. Again, no other OS provides this level of portability. Given a CPU and a hardware platform based/built upon it, you can reasonably expect that Linux runs on it or that someone else has gone through a similar porting process and can assist you in your efforts. You can also expect that the software you write on one Linux architecture can be easily ported to another architecture Linux runs on. There are even device drivers that run on different hardware architectures transparently.
Communication protocol and software standards
Linux also provides broad communication protocol and software standards support, as you’ll see throughout this book. This makes it easy to integrate Linux within existing frameworks and port legacy software to Linux. As such, one can easily integrate a Linux system within an existing Windows network and expect it to serve clients through Samba (using Active Directory or NT-style Primary Domain Controller capabilities), while clients see little difference between it and an NT/Windows 2000 server. You can also use a Linux box to practice amateur radio by building this feature into the kernel, interface with a Bluetooth-enabled cell phone, or roam transparently between a variety of WiFi networks. The OLPC project uses a Linux-based device supporting the latest WiFi mesh networking (yet to be formally standardized at the time of this writing) to enable its laptop units to form self-assembling mesh networks on the fly.
Linux is also Unix-like, and as such, you can easily port traditional Unix programs to it. In fact, many applications currently bundled with the various distributions were first built and run on commercial Unixes and were later ported to Linux. This includes almost all of the fundamental software provided by the FSF. These days, a lot more software is written for Linux, but it’s still designed with portability in mind—even portability to non-Unix systems, such as those from Microsoft, thanks to compatibility libraries such as Cygwin. Traditional embedded OSes are often very limited when it comes to application portability, providing support only for a limited subset of the protocols and software standards available that were considered relevant at the time the OS was conceived.
Available tools
The variety of tools existing for Linux make it very versatile. If you think of an application you need, chances are others already felt the need for it. It is also likely that someone took the time to write the tool and make it available on the Internet. This is what Linus Torvalds did, after all. You can visit the popular websites Freshmeat (http://www.freshmeat.net) and SourceForge (http://www.sourceforge.net) and browse around to see the variety of tools available. Failing that, there’s always Google.
Community support
Community support is perhaps one of the biggest strengths of Linux. This is where the spirit of the free software and open source community can be felt most. As with application needs, it is likely that someone has encountered the same problems as you in similar circumstances. Often, this person will gladly share his solution with you, provided you ask. The development and support mailing lists are the best place to find this community support, and the level of expertise found there often surpasses what can be found through expensive support phone calls with proprietary OS vendors. Usually, when you call a technical support line, you never get to talk to the engineers who built the software you are using. With Linux, an email to the appropriate mailing list will often get you straight to the person who wrote the software. Pointing out a bug and obtaining a fix or suggestions is thereafter a rapid process. As many programmers experience, seldom is a justified plea for help ignored, provided the sender takes the care to search through the archives to ensure that her question hasn’t already been answered.
Licensing
Licensing enables programmers to do with Linux what they could only dream of doing with proprietary software. In essence, you can use, modify, and redistribute the software with only the restriction of providing the same rights to your recipients. This, though, is a simplification of the various licenses used with Linux (GPL, LGPL, BSD, MPL, etc.) and does not imply that you lose control of the copyrights and patents embodied in the software you generate. These considerations will be discussed later in this chapter in Copyright and Patent Issues.” Nonetheless, the degree of liberty available is actually quite large.
Vendor independence
Vendor independence means that you do not need to rely on any sole vendor to get Linux or to use it. Furthermore, if you are displeased with a vendor, you can switch, because the licenses under which Linux is distributed provide you the same rights as the vendors. Some vendors, though, provide additional software in their distributions that isn’t open source, and you might not be able to receive service for this type of software from other vendors. Such issues must be taken into account when choosing distribution. Mostly, though, you can do with Linux as you could do with a car. Since the hood isn’t welded shut, as it is with proprietary software, you can decide to get service from a mechanic other than the one provided by the dealership where you purchased it.
Cost
The cost of Linux is a result of open source licensing and is different from what can be found with traditional embedded OSes. There are three components of software cost in building a traditional embedded system: initial development setup, additional tools, and runtime royalties. The initial development setup cost comprises the purchase of development licenses from the OS vendor. Often, these licenses are purchased for a given number of “seats,” one for each developer. In addition, you may find the tools provided with this basic development package to be insufficient and may want to purchase additional tools from the vendor. This is another cost. Finally, when you deploy your system, the vendor will ask for a per-unit royalty. This may be minimal or large, depending on the type of device you produce and the quantities produced. Some mobile phone manufacturers, for instance, choose to implement their own OSes to avoid paying any royalties. This makes sense for them, given the number of units sold and the associated profit margins.
With Linux, this cost model is turned on its head. Most development tools and OS components are available free of charge, and the licenses under which they are typically distributed prevent the collection of any royalties on these core components. Most developers, though, may not want to go chasing down the various software tools and components and figure out which versions are compatible and which aren’t. Most developers prefer to use a packaged distribution. This involves purchasing the distribution, or it may involve a simple download. In this scenario, vendors provide support for their distribution for a fee and offer services for porting their distributions to new architectures and developing new drivers, also for a fee. Vendors make their money through provision of these services, as well as through additional proprietary software packaged with their distributions. Some vendors do now have a variant of the per-unit royalty (usually termed a “shared risk,” or similar approach), but it is not strictly the same as for those proprietary embedded OSes mentioned before—there’s always a way to use Linux without paying a runtime fee.
Players in the Embedded Linux Scene
Unlike proprietary OSes, Linux is not controlled by a single authority who dictates its future, its philosophy, and its adoption of one technology or another. These issues and others are taken care of by a broad ensemble of players with different but complementary vocations and goals.
Free software and open source community
The free software and open source community is the basis of all Linux development and is the most important player in the embedded Linux arena. It is made up of all of the developers who enhance, maintain, and support the various software components that make up a Linux system. There is no central authority within this group (though there are obvious figureheads). Rather, there is a loosely tied group of independent individuals, each with his specialty. These folks can be found discussing technical issues on the mailing lists concerning them or at gatherings such as the [Ottawa] Linux Symposium. It would be hard to characterize these individuals as a homogeneous group, because they come from different backgrounds and have different affiliations. Mostly, though, they care a great deal about the technical quality of the software they produce. The quality and reliability of Linux, as discussed earlier, are a result of this level of care.
Note that, although many of these developers are affiliated with a given company, their involvement typically goes beyond company lines. They may move from one company to another, but the core developers will always maintain their involvement, no matter who is currently paying their salary. Throughout this book, we will describe quite a few components that are used in Linux systems. Each maintainer of or contributor to the components described herein is considered a player in the free software and open source community.
Industry
Having recognized the potential of Linux in the embedded market, many companies have moved to embrace and promote Linux in this area. Industry players are important because they are the ones pushing Linux as an end-user product. Often, they are the first to receive feedback from those end users. Although postings on the various mailing lists can tell the developer how the software is being used, not all users participate in those mailing lists. Vendors must therefore strike an equilibrium between assisting their users and helping in the development of the various projects making up Linux, without falling into the trap of wanting to divert development to their own ends. In this regard, many vendors have successfully positioned themselves in the embedded Linux market.
Here are some of the better known vendors.
Tip
The vendors listed here are mentioned for discussion purposes only. Neither the authors nor the publisher have evaluated the services provided by any of these vendors for the purposes of this book, and therefore this list should not be interpreted as any form of endorsement.
- MontaVista
Founded by Jim Ready, an embedded industry veteran, and named after a part of the town in which he lived at the time, MontaVista has positioned itself as a leader in the embedded Linux market through its products, services, and promotion of Linux in industrial applications. It produces a variety of products bearing the MontaVista name, including “Professional,” “Carrier Grade,” and “Mobile” varients. MontaVista has contributed to some open source projects, including scheduler enhancements and real-time extensions to the kernel, ViewML, Microwindows, and Linux Trace Toolkit (LTT). A little late converting from the 2.4 kernel over to the 2.6 kernel, MontaVista made up for this by being the first embedded vendor to ship a product featuring the real-time patches to the Linux kernel (this isn’t RTLinux; see Chapter 14). It also makes various claims about capability, and have recently seen considerable success in the cell phone marketplace, especially with Motorola basing entire product families on their MontaVista MobiLinux product.
MontaVista has suffered a little from being the poster child of the embedded Linux revolution. It has seen a number of engineers splinter off and create smaller consultancies—Embedded Alley is one classic example, founded by a number of extremely knowledgeable ex-MontaVista folks—and changes in corporate direction as they decide where the market will take them. MontaVista does not maintain a public repository of its community code contributions (obviously it has developers who work on upstream projects), but it does have the http://source.mvista.com/ website with some public-facing information about projects, such as its real-time initiatives. The primary MontaVista website lives at http://www.mvista.com/.
- Wind River
A relatively latecomer to the embedded Linux scene, Wind River has a long history as author of the proprietary vxworks real-time OS. And this means it has a large customer base behind it, ranging from set top box vendors, to automotive companies, to “deeply embedded” applications (some very small-scale systems that Linux isn’t suited for), to Mars rover robots launched by NASA. After a number of years testing the waters, Wind River finally decided to take the plunge and released an Eclipse-based development product supporting either vxworks or Linux as a target (a great migration tool for existing vxworks developers). The Wind River Linux Center includes various downloads and information, including more detail on its commitment to “Device Software Optimization” (DSO), a term it recently helped to coin. Generally, this is a reference to embedded operating systems such as Linux being more than just the sum of their (Free Software) components, and instead systems that need careful tuning and knowledge to make them useful in a given embedded product.
Wind recently acquired the technology of RTLinux from FSM Labs, and so it is expected to have a number of real-time Linux product offerings by the time you read this. You can find out more about Wind River at http://www.windriver.com/.
- LynuxWorks
This used to be known as Lynx Real-Time Systems and is another one of the traditional embedded OS vendors. Contrary to other traditional embedded OS providers, Lynx decided to embrace Linux early and changed its name to reflect its decision. That, combined with the later acquisition of BSDi by Wind River[4] and QNX’s decision to make its OS available for free download, indicated that open source in general, and Linux in particular, were making serious inroads in the embedded arena. That said, LynuxWorks still develops, distributes, and supports Lynx OS. In fact, LynuxWorks promotes a twofold solution. According to LynuxWorks, programmers needing hard real-time performance should continue to use Lynx, and those who want open source solutions should use BlueCat, its embedded Linux distribution (indeed, they have drawn some criticism for using anti-GPL-like tactics to advocate the use of Lynx OS over Linux in the past). LynuxWorks has even modified its Lynx OS to enable unmodified Linux binaries to run as-is. The fact that LynuxWorks was already a successful embedded OS vendor and that it adopted Linux early confirms the importance of the move toward open source OSes in the embedded market.
- Timesys
Timesys has shifted away from producing a single one-size-fits-all embedded Linux distribution toward a software service model (DSO-like), specializing in custom-built, web-based, cross-compiled packages meeting a range of requirements. Its LinuxLink subscription service is aimed at providing a simple online experience for customizing a selection of required software, having it build automatically for a wide range of targets, and providing a package that can be used on a target device. It claims that it can remove the uncertainty and the hassle of figuring out patches, versions, and dependencies by scripting and automating the process of building custom distributions on the fly. You can find out more at http://www.timesys.com/.
There are also a vast number of smaller players (and more all the time) who provide a variety of services around open source and free software for embedded device application. In fact, many open source and free software contributions are made by individuals who are either independent or work for small-size vendors. As such, the services provided by such small players are often on a par or sometimes surpass those provided by larger players. For example, Wolfgang Denk’s DENX software is a small consultancy based outside of Munich, Germany, yet almost everyone in the embedded Linux space has heard of Wolfgang, his Das U-Boot firmware, or the extensive documentation provided as part of his company’s free Embedded Linux Development Kit (ELDK). Thanks in part to the first edition of this book, vast numbers of embedded Linux developers also know of Karim Yaghmour and his Opersys consultancy.
Resources
Most developers connect to the embedded Linux world through various resource sites and publications. It is through these sites and publications that the Linux development community, industry, and organizations publicize their work and learn about the work of the other players. In essence, the resource sites and publications are the meeting place for all the people concerned with embedded Linux. Two resources stand out: LinuxDevices.com and magazines such as Linux Journal.
LinuxDevices.com was founded on Halloween day[5] 1999 by Rick Lehrbaum (related to one of the MontaVista founders). LinuxDevices.com features news items, articles, polls, forums, and many other links pertaining to embedded Linux. Many key announcements regarding embedded Linux are made on this site, and it contains an archive of actively maintained articles regarding embedded Linux. Though its vocation is clearly commercial, we definitely recommend taking a peek at the site once in a while to keep yourself up-to-date with the latest in embedded Linux (and with their weekly email newsletter, it’s easy to do this). Among other things, LinuxDevices.com was instrumental in launching the Embedded Linux Consortium.
As part of the growing interest in the use of Linux in embedded systems, the Embedded Linux Journal (ELJ) was launched by Specialized System Consultants, owners of Linux Journal (LJ), in January 2001 with the aim of serving the embedded Linux community, but was later discontinued. Though ELJ is no longer published as a separate magazine, it was instrumental in encouraging other Linux magazines to get involved. Several Linux magazines now run embedded features on a regular basis. Indeed, one of the authors of this book was responsible for such a column for a number of years and still writes articles on occasion.
Copyright and Patent Issues
You may ask: what about using Linux in my design? Isn’t Linux distributed under some crazy license that may endanger the copyrights and patents of my company? What are all those licenses anyway? Is there more than one license to take care of? Are we allowed to distribute binary-only kernel modules to protect our IP? What about all these articles I read in the press, some even calling Linux’s license a “virus”?
These questions and many more have probably crossed your mind. You have probably even discussed some of these issues with your coworkers. The issues can be confusing and can come back to haunt you if they aren’t dealt with properly. We don’t say this to scare you. The issues are real, but there are known ways to use Linux without any fear of any sort of licensing contamination. With all the explanations provided next, it is important to keep in mind that this isn’t legal counsel and we are not qualified lawyers. If you have any doubts about your specific project, consult your company attorneys—that’s what they’re there for. Seriously, you want to figure this out now so that it’s not a problem for you later; with a little understanding and forethought, it won’t be.
Textbook GPL
For most components making up a Linux system, there are two licenses involved, the GPL and the LGPL, introduced earlier. Both licenses are available from the FSF’s website at http://www.gnu.org/licenses/ and should be included with any package distributed under the terms of these licenses.[6] The GPL is mainly used for applications, whereas the LGPL is mainly used for libraries. The kernel, the binary utilities, the gcc compiler, and the gdb debugger are all licensed under the GPL. The C library and the GTK widget toolkit, on the other hand, are licensed under the LGPL.
Some programs may be licensed under BSD, Mozilla, or another, but the GPL and LGPL are the main licenses used, but regardless of which one you use, common sense should prevail. Make sure you know the licenses under which the components you use fall and understand their implications. Also make sure you understand the “compatibility” of the licenses for different components that you may wish to use within the same project. Your attorney will be able to advise.
The GPL provides rights and imposes obligations very different from what may be found in typical software licenses. In essence, the GPL is meant to provide a higher degree of freedom to developers and users, enabling them to use, modify, and distribute software with few restrictions. It also makes provisions to ensure that these rights are not abrogated or hijacked in any fashion. To do so, the GPL stipulates the following:
You may make as many copies of the program as you like, as long as you keep the license and copyright intact.
Software licensed under the GPL comes with no warranty whatsoever, unless it is offered by the distributor.
You can charge for the act of copying and for warranty protection.
You can distribute binary copies of the program, as long as you accompany them with the source code used to create the binaries, often referred to as the “original” source code.[7]
You cannot place further restrictions on your recipients than what is specified by the GPL and the software’s original authors.
You can modify the program and redistribute your modifications as long as you provide to your recipients the same rights you received. In effect, any code that modifies or includes GPL code, or any portion of a GPL’d program, cannot be distributed outside your organization under any license other than the GPL. This is the clause some PR folks refer to as being “virus”-like. Keep in mind, though, that this restriction concerns source code only. Packaging the unmodified software for the purpose of running it, as you’ll see, is not subject to this provision.
As you can see, the GPL protects authors’ copyrights while providing freedom of use. This is fairly well accepted. The application of the modification and distribution clauses, on the other hand, generates a fair amount of confusion. To c, two issues must be explained: running GPL software and modifying GPL software. Running the software is usually the reason why the original authors wrote it. The authors of gcc, for example, wrote it for compiling software. As such, the software compiled by an unmodified gcc is not covered by the GPL, since the person compiling the program is only running gcc. In fact, you can compile proprietary software with gcc, and people have been doing this for years, without any fear of GPL “contamination.” Modifying the software, in contrast, creates a derived work that is based on the original software, and is therefore subject to the licensing of that original software. If you take the gcc compiler and modify it to compile a new programming language of your vintage, for example, your new compiler is a derived work and all modifications you make cannot be distributed outside your organization under the terms of any license other than the GPL.
Most anti-GPL speeches or writings play on the confusion between running and modifying GPL software, to give the audience an impression that any software in contact with GPL software is under threat of GPL “contamination.” This is not the case.
There is a clear difference between running and modifying software. As a developer, you can safeguard yourself from any trouble by asking yourself whether you are simply running the software as it is supposed to be run or modifying the software for your own ends. As a developer, you should be fairly capable of making out the difference.
Note that the copyright law makes no difference between static and dynamic linking. Even if your proprietary application is integrated to the GPL software during runtime through dynamic linking, that doesn’t exclude it from falling under the GPL. A derived work combining GPL software and non-GPL software through any form of linking still cannot be distributed under any license other than the GPL. If you package gcc as a dynamic linking library and write your new compiler using this library, you will still be restricted from distributing your new compiler under any license other than the GPL. Some people have attempted to work around dynamic linking restrictions through cunning use of pipes, Unix IPC sockets, and other IPC/RPC protocols to integrate GPL software with their non-GPL product. Depending upon how it is done, such use might be acceptable, but it’s probably not worth the trouble to try working around the GPL in this fashion within your own projects.
Whereas the GPL doesn’t allow you to include parts of the program in your own program unless your program is distributed under the terms of the GPL, the LGPL allows you to use unmodified portions of the LGPL program in your program without any problem. If you modify the LGPL program, though, you fall under the same restrictions as the GPL and cannot distribute your modifications outside your organization under any license other than the LGPL. Linking a proprietary application, statically or dynamically, with the C library, which is distributed under the LGPL, is perfectly acceptable. If you modify the C library, on the other hand, you are prohibited from distributing all modifications (to the library itself) under any license other than the LGPL.
Tip
When you distribute a proprietary application that is linked against LGPL software, you must allow for this LGPL software to be replaced. If you are dynamically linking against a library, for example, this is fairly simple, because the recipient of your software need only modify the library to which your application is linked at startup. If you are statically linking against LGPL software, however, you must also provide your recipient with the object code of your application before it was linked so that she can substitute the LGPL software.
Much like the running versus modifying GPL software discussion earlier, there is a clear difference between linking against LGPL software and modifying LGPL software. You are free to distribute your software under any license when it is linked against an LGPL library. You are not allowed to distribute any modifications to an LGPL library under any license other than LGPL.
Pending issues
Up to now, we have discussed only textbook application of the GPL and LGPL. Some areas of application are, unfortunately, less clearly defined. What about applications that run using the Linux kernel? Aren’t they being linked, in a way, to the kernel’s own code? And what about binary kernel modules, which are even more deeply integrated to the kernel? Do they fall under the GPL? What about including GPL software in my embedded system?
Let us start with the last question. Including a GPL application in your embedded system is actually a textbook case of the GPL. Remember that you are allowed to redistribute binary copies of any GPL software as long as your recipients receive the original source code. Distributing GPL software in an embedded system is a form of binary distribution and is allowed, granted you respect the other provisions of the GPL regarding running and modifying GPL software.
Some proprietary software vendors have tried to cast doubts about the use of GPL software in embedded systems by claiming that the level of coupling found in embedded systems makes it hard to differentiate between applications and, hence, between what falls under GPL and what doesn’t. This is untrue. As we shall see in Chapters 6 and 8, there are known ways to package embedded Linux systems that uphold modularity and the separation of software components.
To avoid any confusion regarding the use of user applications with the Linux kernel, Linus Torvalds has added a preamble to the GPL found with the kernel’s source code. This preamble stipulates that user applications running on the kernel are not subject to the GPL. This means that you can run any sort of application on the Linux kernel without fear of GPL “contamination.” A great number of vendors provide user applications that run on Linux and remain proprietary, including Oracle, IBM, and Adobe.
The area where things have been historically unclear is binary-only kernel modules. Modules are software components that can be dynamically loaded and unloaded to add functionality to the kernel. While they are mainly used for device drivers, they can and have been used for other purposes (for example, for new filesystems, crypto library support for cryptographic storage, and a whole multitude of other purposes). Many components of the kernel can actually be built as loadable modules to reduce the kernel image’s size. When needed, the various modules can be loaded during runtime (as discussed in Chapter 5).
Although this was intended as a facilitating and customizing architecture, many vendors and projects have come to use modules to provide capabilities to the kernel while retaining control over the source code or distributing it under licenses different from the GPL. Some hardware manufacturers, for instance, provide closed-source binary-only module drivers to their users. This enables the use of the hardware with Linux without requiring the vendor to provide details regarding the operation of its device. This is especially true (in the consumer space) when it comes to graphics cards featuring high-end 3D capabilities. In the embedded space, binary modules can range from NAND Flash drivers, to codec support modules, to almost anything else you can think of that someone might consider a valuable piece of intellectual property that they wish to prevent being distributed under the GPL. The authors of this book have seen it all—and so has the Linux community.
The problem is that once a module is loaded in the kernel, it effectively becomes part of the kernel’s address space and is highly coupled to it because of the functions it invokes and the services it provides to the kernel. Because the kernel is itself under the GPL, many contend that modules cannot be distributed under any other license than the GPL because the resulting kernel is a derived work. Others contend that binary-only modules are allowed as long as they use the standard services exported to modules by the kernel. In fact, in response to this logic, the kernel community created wrapped macros EXPORT_SYMBOL, EXPORT_SYMBOL_GPL, and EXPORT_SYMBOL_GPLFUTURE. The idea behind these is that over time, new symbols (kernel functions and data structures) will be exported to the rest of the kernel via one of the GPL macros and thus all symbols will ultimately transition toward being explicitly GPL-only.
For modules already under the GPL, this is obviously a non-issue, but for non-GPL modules, this is a serious issue. Linus has said more than once that he allows binary-only modules as long as it can be shown that the functionality implemented is not Linux-specific (for example, porting a pre-existing graphics driver from Windows to Linux just to make it available for use on Linux systems). Others, however, including Alan Cox and other leading members of the Linux kernel community, have come to question his ability to allow or disallow such modules, because not all the code in the kernel is copyrighted by him. Still others contend that because binary modules have been tolerated for so long, they are part of standard practice.
There is also the case of binary-only modules that use no kernel API whatsoever. The RTAI and RTLinux real-time tasks inserted in the kernel are prime examples. Although it could be argued that these modules are a class of their own and should be treated differently, they are still linked into kernel space and fall under the same rules as ordinary modules, whichever you think them to be.
At the time of this writing, the legal status of binary-only modules has not been tested in court, but there is a growing consensus amongst the Linux kernel community that they are illegal and should not be tolerated. More than one attempt has been made to ban them outright (through technological measures), but the developers involved pointed out that such a technological restriction would make the kernel community no better than those advocating other DRM solutions, which Linux users generally abhor. This issue won’t go away any time soon. In fact, it generally comes up on a semi-annual basis when it appears for a brief moment that binary modules will finally be killed off, before the issue dies down once again. To save a great deal of headache, you are advised to consider strongly whether you really need to have binary kernel modules in the first place. Consult your legal counsel if you are in any way unsure of how to proceed; we can’t tell you what the law says (only how the community will react to you).
One final issue of concern to many is the GPL version 3, which is in the early stages of adoption at the time of this writing. Version 3 updates the previous GPL version 2 from more than a decade ago and includes (ominous-sounding) provisions concerning patents and intellectual property. The goal is, apparently, squarely aimed at embedded developers in an effort to prevent GPL circumvention by means of patent or DRM. Indeed, the phrase “anti-TiVoization” has been applied (TiVo is a set-top box running a modified Linux kernel that uses cryptographic hashes in order to prevent users from replacing the software with their own customized versions). To Richard Stallman, the use of GPL software is undermined whenever an embedded developer introduces cryptographic or DRM measures that effectively prevent users from changing the system, even if the source code is available—a kind of loophole in version 2 that needs some closure. Of course, many are very unhappy at the prospect of making the GPL more militant in this fashion, and a large number of projects have already stated they have no intention of making the switch to version 3. This includes the Linux kernel (which, like many projects, could not convert anyway as it has too many contributors who would need to agree, some of whom have died in the interim). Other projects, such as BusyBox, have expressed discontentment.
We can’t advise you on how version 3 of the GPL might affect your own efforts, but we do recommend, again, that you consult with your company attorney (or your vendor) if you are unsure about its impact.
RTLinux patent
Perhaps one of the most restrictive and controversial licenses you will encounter in deploying Linux in an embedded system is the license to the RTLinux patent held by Victor Yodaiken, the RTLinux project leader. The patent covers the addition of real-time support to general-purpose operating systems as implemented by RTLinux. This patent was recently acquired as part of the Wind River’s purchase of RTLinux technology. At the time of this writing, its use and enforcement in the future is unclear.
Although many have questioned the patent’s viability, given prior art, and Victor’s handling of the issue, it remains that both the patent and the license are currently legally valid, at least in the United States, and have to be accounted for. The U.S. Patent Number for the RTLinux patent is 5,995,745, and you can obtain a copy of it through the appropriate channels. You can read more about the impact of the RTLinux patent on real-time Linux efforts and how they have changed direction in Chapter 12.
A Word on Distributions
Wouldn’t it be simpler and faster to use a distribution instead of setting up your own development environment and building the whole target system from scratch? What’s the best distribution? Unfortunately, there are no straightforward answers to these questions. There are, however, some aspects of distribution use that might help you find answers to these and similar questions.
To use or not to use
First and foremost, you should be aware that it isn’t necessary to use any form of distribution to build an embedded Linux system. In fact, all the necessary software packages are readily available for download on the Internet, and it is these same packages that distribution providers download and package for you to use. This approach provides you with the highest level of control over and understanding of the packages you use and their interactions. Apart from this being the most thorough approach and the one used within this book, it is also the most time-consuming, as you have to take the time to find matching package versions and then set up each package one by one while ensuring that you meet package interaction requirements.
Therefore, if you need a high degree of control over the content of your system, the “do it yourself” method may be best. If, however, like most people, you need the project ready yesterday or if you do not want to have to maintain your own packages, you should seriously consider using both a development and a target distribution. In that case, you will need to choose the development and target distributions most appropriate for you.
How to choose a distribution
Every embedded Linux distribution has its own benefits, so it is difficult to make generalizations about the best one to use for your application. Depending on your project, you may also have other criteria not discussed in this book. In any case, if you choose commercial distributions, make sure you insist upon an evaluation and that you have clear answers to your questions from the distribution vendor before you make any subsequent purchasing decision. Know what kind of support is available to you, what the terms of use and the various licenses are, and how this will affect you. Several vendors (including MontaVista) have developed “shared risk” approaches where you can get discounts in return for subsequent payments. These are not termed royalties per se, but they have some similarities. Know what you are getting yourself into before you commit to anything.
As in any situation, if you ask broad questions, you will get broad answers. Use detailed questions and expect detailed answers. For example, don’t ask whether the Linux kernel you are getting supports real-time applications; instead ask for precise figures, and understand what exactly is being guaranteed to you ahead of time. Make yourself a shopping list of features (and packages) that you would like to see from your chosen distribution and ask to know precisely what is being provided. Do you need to pay more to get additional packages and features? Unclear answers to precise questions are usually a sign that something is amiss. If the vendor (that is trying to do a sale) is unable to answer your questions before you buy the product, do you really expect it to be any different afterward?
Should you instead choose an open source distribution,[8] make sure you have as much information as possible about it. The difference between choosing an open source distribution and a commercial distribution is the way you obtain answers to your questions about the distribution. Whereas the commercial distribution vendor will provide you with answers to your questions about its product, you may have to look for the answers to those same questions about an open source distribution on your own.
An initial factor in the choice of a development or target distribution is the license or licenses involved. Some commercial distributions are partly open source and distribute value-added packages under conventional software licenses that prohibit copying and impose royalties (a form of targeted lock-in). Make sure the distribution clearly states the licenses governing the usage of the value-added software and their applicability. If unsure, ask. Don’t leave licensing issues unclear. This will only serve to cause you undue pain should you ever decide to migrate away to a different embedded Linux distribution.
One thing that distinguishes commercial distributions from open source distributions is the support provided by the vendor. Whereas the vendor supplying a commercial distribution almost always provides support for its own distribution, the open source community supplying an open source distribution does not necessarily provide the same level of support that would be expected from a commercial vendor. This does not preclude some vendors from providing commercial support for open source distributions. Through serving different customers with different needs in the embedded field, the various vendors build a unique knowledge about the distributions they support and the problems clients might encounter during their use, and are therefore best placed to help you efficiently. Mainly, though, these vendors are the ones that keep up with the latest and greatest in Linux and are therefore the best source of information regarding possible bugs and interoperability problems that may show up.
Reputation can also come into play when choosing a distribution, but it has to be used wisely, as a lot of information circulating may be presented as fact but instead be mere interpretation. If you’ve heard something about one distribution or another, take the time to verify the validity of the information. In the case of a commercial distribution, contact the vendor. Chances are it knows where this information comes from and, most importantly, the rational explanation for it. This verification process, though, isn’t specific to embedded Linux distributions, but what is specific is the reputation commercial distributions build when their vendors contribute to the open source community. A vendor that gives back by providing more open source software or by financing development shows that it is in contact with the open source community and therefore understands how the changes and developments of the various open source projects will affect its future products and, ultimately, its clients. In short, this is a critical link and a testament to the vendor’s understanding of the dynamics involved in the development of the software it provides you. In the case of open source distributions, this criterion is already met, as the distribution itself is an open source contribution.
Another precious tool that commercial distributions might have to offer is documentation. In this day and age where everything is ever-changing, up-to-date and accurate documentation is a rare commodity. The documentation for the majority of open source projects is often out-of-date, if available at all. Linus Torvalds’s words ring true here: “Use the source, Luke,” meaning that if you need to understand the software you should read the source code. Yet not everyone can invest the amount of time necessary to achieve this level of mastery, hence the need for appropriate documentation. Because the open source developers prefer to invest more time in writing code than in writing documentation, it is up to the distribution vendors to provide appropriately packaged documentation with their distributions. When evaluating a distribution, make sure to know the type and extent of accompanying documentation. Although there is less documentation for open source distributions in comparison with commercial distributions, some open source distributions are remarkably well documented.
Given the complexity of some aspects of development and target setup, the installation of a development and/or target distribution can be difficult. In this regard, you may be looking for easy-to-install distributions. Although this is legitimate, keep in mind that once you’ve installed the distributions, you should not need to reinstall them afterward. Notice also that installation does not really apply for a target distribution as it was defined earlier, because target distributions are used to facilitate the generation of target setups and don’t have what is conventionally known as an “installation” process. The three things you should look for in the installation process of a distribution are clear explanations (whether textually during the installation, in a manual, or both), configurability, and automation. Configurability is a measure of how much control you have over the packages being installed, whereas automation is the ability to automate the process using files containing the selected configuration options.
With some CPU models and boards being broadly adopted for embedded systems development, commercial distribution vendors have come to provide prepackaged development and/or target distributions specifically tailored for those popular CPU models and boards. If you intend to use a specific CPU model or board, you may want to look for a distribution that is already tested for your setup.
What to avoid doing with a distribution
There is one main course of action to avoid when using a distribution: using the distribution in a way that makes you dependent solely on this same distribution for all future development. Remember that one of the main reasons to use Linux is that you aren’t subject to anyone’s will or market decisions. If your development relies only on proprietary tools and methods of the distribution you chose, you risk being locked into continuous use of that same distribution for all future development. This does not mean, though, that you shouldn’t use commercial distributions with value-added software that cannot be found on other distributions. It only means that you should have a backup plan to achieve the same results with different tools from different distributions, just in case. Many embedded vendors have already standardized on development tools such as Eclipse—with each vendor adding slightly different “value-add” plug-ins—and use of such tools should serve to minimize the disruption to your engineering efforts if you ever have to switch to a different Eclipse-based tool.
Design and Implementation Methodology
Designing and implementing an embedded Linux system can be carried out in a defined manner. The process includes many tasks, some of which may be carried out in parallel, thereby reducing overall development time. Some tasks can even be omitted if a distribution is being used. Regardless of the actual tools or methodology you use, Chapter 2 is required reading for all tasks involved in building an embedded Linux system.
Creating a Target Linux System
A target Linux system is created by configuring and bundling together the appropriate system components. Programming and development aspects are a separate subject, and they are discussed later in this chapter.
There are four main steps to creating a target Linux system:
Determining system components is like making a shopping list before you go to the grocery store. It is easy to go without a shopping list and wonder at all the choices you have, as many do with Linux. This may result in “featurism,” whereby your system will have lots and lots of features but won’t necessarily fulfill its primary purpose. Therefore, before you go looking at all the latest Linux gizmos, sit down and write a list of what you need. We find that this approach helps in focusing development and avoids distractions like “Look, honey, they actually have salami ice cream!” This doesn’t mean that you shouldn’t change your list if you see something pertinent; it is just a warning about the quantity of software available for Linux and the inherent abundance of choices.
Chapter 3 discusses the hardware components that can be found as part of an embedded Linux system. It should provide you with enough background and maybe even ideas of what hardware you can find in an embedded Linux system. As Linux and surrounding software are ever-evolving targets, use this and further research on the Net to find out which design requirements Linux meets. In turn, this will provide you with a list of items you need to develop in order to complete your system. This step of development is the only one that cannot be paralleled with other tasks. Determining system requirements and Linux’s compliance to these requirements has to be completed before any other step.
Because of the evolving nature of Linux, you may feel the need to get the latest and greatest pieces of software for your design. Avoid doing this, as new software often needs testing and may require upgrades to other software because of the dependencies involved between packages. Hence, you may find yourself locked in a frantic race to keep up with the plethora of updates. Instead, fix the bugs with the current software you have and keep track of other advances so that the next generation projects you design can profit from these advances. If you have an important reason to upgrade a software component, carefully analyze the consequences of such an upgrade on the rest of your system before actually carrying it out. You may also want to try the upgrade on a test system before applying it to your main system.
Having determined which features are pertinent to your design, you can select a kernel version and relevant configuration. Chapter 5 covers the configuration and build process of the kernel. Unlike other pieces of software, you may want to keep updating your kernel to the latest stable version throughout your project’s development, up until the beta stage. Though keeping the kernel version stable throughout the development cycle may seem simple, you might find yourself trying to fix bugs that have been fixed in more recent kernels. Keeping yourself up-to-date with recent kernel developments, as we discuss in Chapter 5, will help you decide whether updating to the most recent kernel is best for you. Also, you may want to try newer kernels and roll back to older ones if you encounter any serious problems. Note that using kernels that are too old may cut you off from community support, since contributors can rarely afford to keep answering questions about old bugs.
While we do encourage you to keep up-to-date, it is worth mentioning major changes to the kernel, the kind that happen every few years. Consider how the 2.4 series kernel remains in use in embedded designs even after the 2.6 kernel has long since proven itself. This isn’t an accident; it happened because engineers became comfortable with the 2.4 kernel and felt no need to make the switch for their existing products and embedded designs. This doesn’t mean you should use the 2.4 kernel in your new design, but it does mean that you should carefully consider the impact of major version changes of any software—including the Linux kernel. It’s one thing to upgrade from kernel 2.6.20 to 2.6.21, but quite another to migrate from one major release to the next. Treat that kind of transition as you would any other major software component upgrade, especially if you have a series of product-specific modifications to forward port over to the newer kernel when you make the transition.
Regardless of whether you decide to follow kernel updates, we suggest you keep the kernel configuration constant throughout the project. This will avoid completed parts from breaking in the course of development. This involves studying the configuration options closely, though, in light of system requirements. Although this task can be conducted in parallel with other tasks, it is important that developers involved in the project be aware of the possible configuration options and agree with the options chosen.
Once configuration is determined, it is time to build the kernel. Building the kernel involves many steps and generates more than just a kernel image. Although the generated components are not necessary for some of the other development aspects of the project, the other project components tend to become more and more dependent on the availability of the kernel components as the project advances. It is therefore preferable to have the kernel components fully configured and built as early as possible, and kept up-to-date throughout the project.
In parallel to handling the kernel issues, you can start building the root filesystem of the embedded system, as explained in Chapter 6. The root filesystem of an embedded Linux system is similar to the one you find on a workstation or server running Linux, except that it contains only the minimal set of applications, libraries, and related files needed to run the system. Note that you should not have to remove any of the components you previously chose at this stage to obtain a properly sized root filesystem. In fact, if you have to do so, you probably did not determine system components adequately. Remember that this earlier stage should include an analysis of all system requirements, including the root filesystem size. You should therefore have as accurate an estimate as possible of the size of each component you selected during the first step of creating the target system.
If you are unable to predetermine the complete list of components you will need in your embedded system and would rather build your target root filesystem iteratively by adding the tools and libraries you need as you go along, then do so, but do not treat the result as your final root filesystem. Instead, use the iterative method to explore building root filesystems, and then apply your experience to building a clean root filesystem for your target system. The reason behind this is that the trial-and-error nature of the iterative method makes its completion time nondeterministic. The structured approach may require more forethought, but its results are known and can be the basis for additional planning.
Setting up and configuring the storage devices and the bootloader software are the remaining tasks in creating a target Linux system. Chapters 7, 8, and 9 discuss these issues in full. It is during these steps that the different components of the target system come together: the bootloader, the root filesystem, and the kernel. As booting is highly dependent on the architecture, different bootloaders are involved. Within a single architecture there are also variations in the degree of debugging and monitoring provided by the bootloaders. The methodology to package and boot a system is fairly similar among the different architectures, but varies according to the permanent storage device from which the system is booted and which bootloader is used. Booting a system from native flash, for instance, is different than booting a system from a SATA disk device or CompactFlash device, and is even more different than booting from a network server.
Setting Up and Using Development Tools
Software development for embedded systems is different from software development for the workstation or server environments. Mainly, the target environment is often dissimilar to the host on which the development is conducted. Hence the need for a host/target setup whereby the developer develops his software on the host and downloads it onto the target for testing. There are two aspects to this setup: development and debugging. Such a setup, however, does not preclude you from using Linux’s multi-architecture advantage to test your target’s applications on your host with little or no modification. Though not all applications can be tested in this way, testing target applications on the host will generally save you a lot of time.
Embedded development is discussed in Chapter 4. Prior to testing any code on the target system, it is necessary to establish a host/target connection. This will be the umbilical cord by which the developer will be able to interact with the target system to verify whether the applications he develops function as prescribed. As the applications cannot typically run on bare hardware, there will have to be a functional embedded Linux system on the target hardware. Since it is often impossible to wait for the final target setup to be completed to test target applications, you can use a development target setup. The latter will be packaged much more loosely and will not have to respect the size requirements imposed on the final package. Hence, the development root filesystem may include many more applications and libraries than will be found in the final root filesystem. This also allows different and larger types of permanent storage devices during development.
Obtaining such a setup necessitates compiling the target applications and libraries. This is achieved by configuring or building the various compiler and binary utilities for cross-development. Using these utilities, you can build applications for the target and therefore build the development target setup used for further development. With this done, you can use various integrated development environments (IDEs) to ease development of the project components, and use other tools such as CVS, Subversion, and GIT to coordinate work among developers.
Given the horsepower found on some embedded systems, some developers even choose to carry out all development directly on the target system. In this setup, the compiler and related tools all run on the target. This, in effect, combines host and target in a single machine and resembles a conventional workstation application development. The main advantage of such a configuration is that you avoid the hassle of setting up a host/target environment.
Whatever development setup you choose, you will need to
debug and poke at your software in many ways. You can do this with the
debugging tools covered in Chapter 11. For
simple debugging operations, you may choose to use ad hoc methods such
as printing values using printf().
Some problems require more insight into the runtime operations of the
software being debugged; this may be provided by symbolic debugging. gdb is the
most common general-purpose debugger for Linux, but symbolic debugging
on embedded systems may be more elaborate. It could involve such things
as remote serial debugging, kernel debugging, and BDM and JTAG debugging
tools. But even symbolic debugging may be inadequate in some situations.
When system calls made by an application are problematic or when
synchronization problems need to be solved, it is better to use tracing
tools such as strace and LTT. For performance
problems, there are other tools more adapted to the task, such as
gprof and gcov. When all else
fails, you may even need to understand kernel crashes.
Developing for the Embedded
One of the main advantages of using Linux as an embedded OS is that the code developed for Linux should run identically on an embedded target and on a workstation, right? Well, not quite. Although it is true that you can expect your Linux workstation code to build and run the same on an embedded Linux system, embedded system operations and requirements differ greatly from workstation or server environments. Whereas you can expect errors to kill an application on a workstation, for instance, leaving the responsibility to the user to restart the application, you can’t afford to have this sort of behavior in an embedded system. Neither can you allow applications to gobble up resources without end or behave in an untimely manner.[9] Therefore, even though the APIs and OS used may be identical, there are fundamental differences in programming philosophies.
Networking
Networking enables an embedded system to interact with and be accessible to the outside world. In an embedded Linux environment, you have to choose networking hardware, networking protocols, and the services to offer while accounting for network security. Chapter 10 covers the setup and use of networking services such as HTTP, Telnet, SSH, and/or SNMP. One interesting feature in a network-enabled embedded system is the possibility of remote updating, whereby it is possible to update the system via a network link without on-site intervention. (This is covered in Chapter 8.)
[3] It would be tempting to call these “host distributions,” but as you’ll see later, some developers choose to develop directly on their target, hence the preference for “development distributions.”
[4] Wind River has since changed its mind, and its relationship with BSD seems to be a thing of the past.
[5] The date was selected purposely in symbolic commemoration of the infamous Halloween Documents uncovered by Eric Raymond. If you are not familiar with these documents and their meaning, have a look at http://www.opensource.org/halloween/.
[6] The licenses are often stored in a file called COPYING, for the GPL, and a file called COPYING.LIB, for the LGPL. Copies of these files are likely to have been installed somewhere on your disk by your distribution.
[7] The specific wording of the GPL to designate this code is the following: “The source code for a work means the preferred form of the work for making modifications to it.” Delivering binaries of an obfuscated version of the original source code to try circumventing the GPL is a trick that has been tried before, and it doesn’t work.
[8] An open source distribution is one that is maintained by the open source community, such as Debian. Inherently, such distributions do not contain any proprietary software.
[9] Normal Linux workstation and server applications should not gobble up resources either. In fact, the most important applications used on Linux servers are noteworthy for their stability, which is one reason Linux is so successful as a server operating system.
Get Building Embedded Linux Systems, 2nd Edition now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.
