Chapter 1. Introduction to Linux

Linux is the most widely used operating system, used in everything from mobile devices to the cloud.

You might not be familiar with the concept of an operating system. Or you might be using an operating system such as Microsoft Windows without giving it too much thought. Or maybe you are new to Linux. To set the scene and get you in the right mindset, we’ll take a bird’s-eye view of operating systems and Linux in this chapter.

We’ll first discuss what modern means in the context of the book. Then we’ll review a high-level Linux backstory, looking at important events and phases over the past 30 years. Further, in this chapter you’ll learn what the role of an operating system is in general and how Linux fills this role. We also take a quick look at what Linux distributions are and what resource visibility means.

If you’re new to operating systems and Linux, you’ll want to read the entire chapter. If you’re already experienced with Linux, you might want to jump to “A Ten-Thousand-Foot View of Linux”, which provides a visual overview as well as mapping to the book’s chapters.

But before we get into the technicalities, let’s first step back a bit and focus on what we mean when we say “modern Linux.” This is, surprisingly, a nontrivial matter.

What Are Modern Environments?

The book title specifies modern, but what does that really mean? Well, in the context of this book, it can mean anything from cloud computing to a Raspberry Pi. In addition, the recent rise of Docker and related innovations in infrastructure has dramatically changed the landscape for developers and infrastructure operators alike.

Let’s take a closer look at some of these modern environments and the prominent role Linux plays in them:

Mobile devices

When I say “mobile phone” to our kids, they say, “In contrast to what?” In all fairness and seriousness, these days many phones—depending on who you ask, up to 80% or more—as well as tablets run Android, which is a Linux variant. These environments have aggressive requirements around power consumption and robustness, as we depend on them on a daily basis. If you’re interested in developing Android apps, consider visiting the Android developer site for more information.

Cloud computing

With the cloud, we see at scale a similar pattern as in the mobile and micro space. There are new, powerful, secure, and energy-saving CPU architectures such as the successful ARM-based AWS Graviton offerings, as well as the established heavy-lifting outsourcing to cloud providers, especially in the context of open source software.

Internet of (Smart) Things

I’m sure you’ve seen a lot of Internet of Things (IoT)–related projects and products, from sensors to drones. Many of us have already been exposed to smart appliances and smart cars. These environments have even more challenging requirements around power consumption than mobile devices. In addition, they might not even be running all the time but, for example, only wake up once a day to transmit some data. Another important aspect of these environments is real-time capabilities. If you’re interested in getting started with Linux in the IoT context, consider the AWS IoT EduKit.

Diversity of processor architectures

For the past 30 years or so, Intel has been the leading CPU manufacturer, dominating the microcomputer and personal computer space. Intel’s x86 architecture was considered the gold standard. The open approach that IBM took (publishing the specifications and enabling others to offer compatible devices) was promising, resulting in x86 clones that also used Intel chips, at least initially.

While Intel is still widely used in desktop and laptop systems, with the rise of mobile devices we’ve seen the increasing uptake of the ARM architecture and recently RISC-V. At the same time, multi-arch programming languages and tooling, such as Go or Rust, are becoming more and more widespread, creating a perfect storm.

All of these environments are examples of what I consider modern environments. And most, if not all of them, run on or use Linux in one form or another.

Now that we know about the modern (hardware) systems, you might wonder how we got here and how Linux came into being.

The Linux Story (So Far)

Linux celebrated its 30th birthday in 2021. With billions of users and thousands of developers, the Linux project is, without doubt, a worldwide (open source) success story. But how did it all this start, and how did we get here?

1990s

We can consider Linus Torvalds’s email on August 25, 1991, to the comp.os.minix newsgroup as the birth of the Linux project, at least in terms of the public record. This hobby project soon took off, both in terms of lines of code (LOC) and in terms of adoption. For example, after less than three years, Linux 1.0.0 was released with over 176,000 LOCs. By that time, the original goal of being able to run most Unix/GNU software was already well reached. Also, the first commercial offering appeared in the 1990s: Red Hat Linux.

2000 to 2010

As a “teenager,” Linux was not only maturing in terms of features and supported hardware but was also growing beyond what UNIX could do. In this time period, we also witnessed a huge and ever-increasing buy-in of Linux by the big players, that is, adoption by Google, Amazon, IBM, and so on. It was also the peak of the distro wars, resulting in businesses changing their directions.

2010s to now

Linux established itself as the workhorse in data centers and the cloud, as well as for any types of IoT devices and phones. In a sense, one can consider the distro wars as being over (nowadays, most commercial systems are either Red Hat or Debian based), and in a sense, the rise of containers (from 2014/15 on) is responsible for this development.

With this super-quick historic review, necessary to set the context and understand the motivation for the scope of this book, we move on to a seemingly innocent question: Why does anyone need Linux, or an operating system at all?

Why an Operating System at All?

Let’s say you do not have an operating system (OS) available or cannot use one for whatever reason. You would then end up doing pretty much everything yourself: memory management, interrupt handling, talking with I/O devices, managing files, configuring and managing the network stack—the list goes on.

Note

Technically speaking, an OS is not strictly needed. There are systems out there that do not have an OS. These are usually embedded systems with a tiny footprint: think of an IoT beacon. They simply do not have the resources available to keep anything else around other than one application. For example, with Rust you can use its Core and Standard Library to run any app on bare metal.

An operating system takes on all this undifferentiated heavy lifting, abstracting away the different hardware components and providing you with a (usually) clean and nicely designed Application Programming Interface (API), such as is the case with the Linux kernel that we will have a closer look at in Chapter 2. We usually call these APIs that an OS exposes system calls, or syscalls for short. Higher-level programming languages such as Go, Rust, Python, or Java build on top of those syscalls, potentially wrapping them in libraries.

All of this allows you to focus on the business logic rather than having to manage the resources yourself, and also takes care of the different hardware you want to run your app on.

Let’s have a look at a concrete example of a syscall. Let’s say we want to identify (and print) the ID of the current user.

First, we look at the Linux syscall getuid(2):

...
getuid() returns the real user ID of the calling process.
...

OK, so this getuid syscall is what we could use programmatically, from a library. We will discuss Linux syscalls in greater detail in “syscalls”.

Note

You might be wondering what the (2) means in getuid(2). It’s a terminology that the man utility (think built-in help pages) uses to indicate the section of the command assigned in man, akin to a postal or country code. This is one example where the Unix legacy is apparent; you can find its origin in the Unix Programmer’s Manual, seventh edition, volume 1 from 1979.

On the command line (shell), we would be using the equivalent id command that in turn uses the getuid syscall:

$ id --user
638114

Now that you have a basic idea of why using an operating system, in most cases, makes sense, let’s move on to the topic of Linux distributions.

Linux Distributions

When we say “Linux,” it might not be immediately clear what we mean. In this book, we will say “Linux kernel,” or just “kernel,” when we mean the set of syscalls and device drivers. Further, when we refer to Linux distributions (or distros, for short), we mean a concrete bundling of kernel and related components, including package management, file system layout, init system, and a shell, preselected for you.

Of course, you could do all of this yourself: you could download and compile the kernel, choose a package manager, and so on, and create (or roll) your own distro. And that’s what many folks did in the beginning. Over the years, people figured out that it is a better use of their time to leave this packaging (and also security patching) to experts, private or commercial, and simply use the resulting Linux distro.

Tip

If you are inclined to build your own distribution, maybe because you are a tinkerer or because you have to due to certain business restrictions, I recommend you take a closer look at Arch Linux, which puts you in control and, with a little effort, allows you to create a very customized Linux distro.

To get a feeling for the vastness of the distro space, including traditional distros (Ubuntu, Red Hat Enterprise Linux [RHEL], CentOS, etc., as discussed in Chapter 6) and modern distros (such as Bottlerocket and Flatcar; see Chapter 9), take a look at DistroWatch.

With the distro topic out of the way, let’s move on to a totally different topic: resources and their visibility and isolation.

Resource Visibility

Linux has had, in good UNIX tradition, a by-default global view on resources. This leads us to the question: what does global view mean (in contrast to what?), and what are said resources?

Note

Why are we talking about resource visibility here in the first place? The main reason is to raise awareness about this topic and to get you in the right state of mind for one of the important themes in the context of modern Linux: containers. Don’t worry if you don’t get all of the details now; we will come back to this topic throughout the book and specifically in Chapter 6, in which we discuss containers and their building blocks in greater detail.

You might have heard the saying that in Unix, and by extension Linux, everything is a file. In the context of this book, we consider resources to be anything that can be used to aid the execution of software. This includes hardware and its abstractions (such as CPU and RAM, files), filesystems, hard disk drives, solid-state drives (SSDs), processes, networking-related stuff like devices or routing tables, and credentials representing users.

Warning

Not all resources in Linux are files or represented through a file interface. However, there are systems out there, such as Plan 9, that take this much further.

Let’s have a look at a concrete example of some Linux resources. First, we want to query a global property (the Linux version) and then specific hardware information about the CPUs in use (output edited to fit space):

$ cat /proc/version 1
Linux version 5.4.0-81-generic (buildd@lgw01-amd64-051)
(gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04))
#91~18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021

$ cat /proc/cpuinfo | grep "model name" 2
model name      : Intel Core Processor (Haswell, no TSX, IBRS)
model name      : Intel Core Processor (Haswell, no TSX, IBRS)
model name      : Intel Core Processor (Haswell, no TSX, IBRS)
model name      : Intel Core Processor (Haswell, no TSX, IBRS)
1

Print the Linux version.

2

Print CPU-related information, filtering for model.

With the preceding commands, we learned that this system has four Intel i7 cores at its disposal. When you log in with a different user, would you expect to see the same number of CPUs?

Let’s consider a different type of resource: files. For example, if the user troy creates a file under /tmp/myfile with permission to do so (“Permissions”), would another user, worf, see the file or even be able to write to it?

Or, take the case of a process, that is, a program in memory that has all the necessary resources available to run, such as CPU and memory. Linux identifies a process using its process ID, or PID for short (“Process Management”):

$ cat /proc/$$/status | head -n6 1
Name:   bash
Umask:  0002
State:  S (sleeping)
Tgid:   2056
Ngid:   0
Pid:    2056
1

Print process status—that is, details about the current process—and limit output to show only the first six lines.

Can there be multiple processes with the same PID in Linux? What may sound like a silly or useless question turns out to be the basis for containers (see “Containers”). The answer is yes, there can be multiple processes with the same PID, in different contexts called namespaces (see “Linux Namespaces”). This happens, for example, in a containerized setup, such as when you’re running your app in Docker or Kubernetes.

Every single process might think that it is special, having PID 1, which in a more traditional setup is reserved for the root of the user space process tree (see “The Linux Startup Process” for more details).

What we can learn from these observations is that there can be a global view on a given resource (two users see a file at the exact same location) as well as a local or virtualized view, such as the process example. This raises the question: is everything in Linux by default global? Spoiler: it’s not. Let’s have a closer look.

Part of the illusion of having multiple users or processes running in parallel is the (restricted) visibility onto resources. The way to provide a local view on (certain supported) resources in Linux is via namespaces (see “Linux Namespaces”).

A second, independent dimension is that of isolation. When I use the term isolation here, I don’t necessarily qualify it—that is, I make no assumptions about how well things are isolated. For example, one way to think about process isolation is to restrict the memory consumption so that one process cannot starve other processes. For example, I give your app 1 GB of RAM to use. If it uses more, it gets out-of-memory killed. This provides a certain level of protection. In Linux we use a kernel feature called cgroups to provide this kind of isolation, and in “Linux cgroups” you will learn more about it.

On the other hand, a fully isolated environment gives the appearance that the app is entirely on its own. For example, a virtual machine (VM; see also “Virtual Machines”) can be used to provide you with full isolation.

A Ten-Thousand-Foot View of Linux

Whoa, we went quite deep into the weeds already. Time to take a deep breath and re-focus. In Figure 1-1, I’ve tried to provide you with a high-level overview of the Linux operating system, mapping it to the book chapters.

lmlx 0101
Figure 1-1. Mapping the Linux operating system to book chapters

At its core, any Linux distro has the kernel, providing the API that everything else builds on. The three core topics of files, networking, and observability follow you everywhere, and you can consider them the most basic building blocks above the kernel. From a pure usage perspective, you will soon learn that you will most often be dealing with the shell (Where is the output file for this app?) and things related to access control (Why does this app crash? Ah, the directory is read-only, doh!).

As an aside: I’ve collected some interesting topics, from virtual machines to modern distros, in Chapter 9. I call these topics “advanced” mainly because I consider them optional. That is, you could get away without learning them. But if you really, really, really want to benefit from the full power that modern Linux can provide you, I strongly recommend that you read Chapter 9. I suppose it goes without saying that, by design, the rest of the book—that is Chapter 2 to Chapter 8—are essential chapters you should most definitely study and apply the content as you go.

Conclusion

When we call something “modern” in the context of this book, we mean using Linux in modern environments, including phones, data centers (of public cloud providers), and embedded systems such as a Raspberry Pi.

In this chapter, I shared a high-level version of the Linux backstory. We discussed the role of an operating system in general—to abstract the underlying hardware and provide a set of basic functions such as process, memory, file, and network management to applications—and how Linux goes about this task, specifically regarding visibility of resources.

The following resources will help you continue getting up to speed as well as dive deeper into concepts discussed in this chapter:

O’Reilly titles
Other resources

And now, without further ado: let’s start our journey into modern Linux with the core, erm, kernel, of the matter!

Get Learning Modern Linux now with the O’Reilly learning platform.

O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Start your free trial