Modern System Administration

Chapter 1. Patterns and Interconnections

Imagine you are making a cake with a friend. You followed the recipe, mixing all the ingredients (oil, flour, eggs, and sugar), and it looks OK, but when you taste it, something isn’t quite right. To be an accomplished baker, you must understand all the elements of a cake (ratios of flour to fat, etc.) and how they work together to impact the finished product’s quality (e.g., taste and texture). For example, in Figure 1-1, our bakers didn’t understand that sesame oil wasn’t an appropriate oil to include in the cake.

Now replace baker with sysadmin, and replace the golden ratios of a baker’s ingredients with the interconnected components in your system (e.g., smartphones, embedded devices, large servers, and storage arrays). To be an accomplished sysadmin, you need to understand how the components connect in common patterns and impact your system’s quality (e.g., reliability, scalability, and maintainability).

In this chapter, I will help you reason about your systems to see the patterns and interconnections in them so that you understand what informed the choices behind your system’s design.

Figure 1-1. Modeling understanding a system (image by Tomomi Imura)

How to Connect Things

Engineers choose architecture patterns, the reusable solutions to address typical workloads (e.g., batch processing, web servers, and caching). These patterns are models that convey “a shared understanding of the system’s design.”¹

From on-prem to cloud computing environments, the reusable solutions are evolving to support decomposed sets of smaller sets of services.² These patterns shape systems’ reliability, scalability, and maintainability by determining the components of the system and the connection between the components.

Let’s examine three common architecture patterns used in system design so you can see how their use informs (and limits) their evolution and system qualities (reliability, scalability, and maintainability):

Layered architecture

The most common and familiar pattern is a general-purpose layered or tiered architecture pattern. Engineers commonly use this pattern for client-server applications like web servers, email, and other business applications.

Engineers organize components into horizontal layers, with each layer performing a specific role, separating the concerns of each layer from the other layers. Layers are generally tightly coupled depending on a request and response to and from the adjacent layers. As a result, you can update and deploy components within each layer independently from other layers.

A two-tier system is composed of a client and a server. A three-tier system comprises a client and two other layers of servers; the presentation, application, and data tiers are abstracted into different components. Each tier may be split into separate logical layers in an N-tier or multitier system. There may be more than three tiers depending on the system’s needs (e.g., resilience, security, and scalability). With each tier, scalability and reliability increase as the tiers create additional separation between concerns that can be deployed and updated independently.

Microservices architecture

A microservice system is a distributed architecture that, instead of tiers, consists of a collection of small units of business code decoupled from one another. Microservices are small and autonomous. Because each service is separate, code can be developed and deployed independently. In addition, each service can leverage the best language or framework for its use case.

Microservices increase the system’s scalability and reliability because they can be independently deployed as needed and isolated from the points of failure in the system.

Decomposing a service into microservices decreases maintainability due to the increased cognitive load on the sysadmins. To understand your system, you need to know all the details about each separate service (i.e., languages, frameworks, build and deploy pipelines, and any relevant environments).

Event-driven architecture

An event-driven architecture is a distributed asynchronous pattern that enables loose coupling between applications. Different applications don’t know details about one another. Instead, they communicate indirectly by publishing and consuming events.

Events are something that happens, a fact that can be tracked.³ Systems generate events. In event-driven systems, event producers create events, brokers ingest events, and consumers listen and process events.

There are two main models for event-driven systems: messaging (or pub/sub) and streaming.

The event producer or publishers publish events to a broker in an event messaging system. The broker sends all posted events to event consumers or subscribers. The message broker receives published events from publishers, maintains the order of received messages, makes them available to subscribers, and deletes events after they are consumed.

In an event streaming system, events are published to a distributed log, a durable append-only data store. As a result, consumers consume events from the stream that they want and can replay events. In addition, the distributed log retains events after they have been consumed, meaning that new subscribers can subscribe to events that occurred before their subscription.

Because components are loosely coupled, individual parts of the system don’t have to worry about the health of other components. Loosely coupled elements increase the resiliency of the overall system as they can be independently deployed and updated. Event persistence enables the replaying of events that occurred in the case of a failure.

Table 1-1 summarizes the relative comparisons of reliability, scalability, and maintainability of the three common architecture patterns you will see in systems.

Table 1-1. Comparison of reliability, scalability, and maintainability of architectures

	Layered	Microservices	Event driven
Reliability	Medium (tightly coupled systems)	High	High
Scalability	Medium (limited within layers)	High	High
Maintainability	High	Low (decreased simplicity)	Medium (decreased testability)

Tip

Of course, these are not the only patterns you’ll see in system design. Check out Martin Fowler’s comprehensive website, the Software Architecture Guide.

How Things Communicate

A component of a system doesn’t exist in isolation—each component will communicate with other components in the system, and that communication may be informed by the architecture pattern (REST for N-tier architecture, gRPC for event-driven architecture).

There a few different models used to represent how components communicate, e.g., the Internet model, five-layer Internet model, TCP/IP five-layer reference model, and TCP/IP model. While these models are very similar, they have slight differences that may inform how people think about the applications and services they build to communicate.

When an individual or group of engineers identify an area for improvement, they author a Request for Comment (RFC) and submit it for peer review. As an open international community that works to maintain and improve the internet’s design, usability, maintainability, and interoperability, the Internet Engineering Task Force (IETF) adopts some of the proposed RFCs as technical standards that define the official specifications and protocols. The protocol specifications define how devices communicate with one another while loosely following the Internet model. And these protocols continue to evolve as the Internet grows and the needs of people change (for an example of this, check out Appendix B).

As depicted in Table 1-2, the five-layer Internet model shows five discrete layers. Each layer communicates via the interfaces above and below via a message object specific to the layer. Layering a system separates the responsibilities at each layer and enables different system parts to be built (and changed). It also allows differentiation at each of the layers.

Table 1-2. Five-layer Internet model and example protocols

Layers	Example protocols
Application	HTTP, DNS, BGP
Transport	TCP, UDP
Network	IP, ICMP
Data Link	ARP
Physical	Copper, optical fiber, WiFi

Like the cake in Figure 1-1, there aren’t any crisp layers that inform you precisely where a problem emerged. Protocol implementations are not required to follow the specifications strictly and overlap layers. For example, the protocol that determines the fastest and most efficient route to transmit data is the Border Gateway Protocol (BGP). Because of the implementation of the protocol, people may classify BGP at either the application layer or the transport layer.

The layers in the Internet model give you a way to frame the context and narrow your focus to the ingredients of your application—the source code and dependencies—or at a lower physical level, simplifying the complex communication model into understandable chunks. However, sometimes you will run into situations where the reduction in context doesn’t help you understand what is happening. To level up your comprehension, you have to know how everything works together to impact your system’s quality.

Learning More About Protocols

If your role entails managing networks or implementations of protocols, check out Andrew Tanenbaum’s Computer Networks (Pearson), Kevin R. Fall and W. Richard Stevens’s TCP/IP Illustrated, Volume 1: The Protocols, 2nd edition (Addison-Wesley), and the website the RFC Series.

Let’s look in more detail at the application, transport, network, data link, and physical layers.

Application Layer

Let’s start at the top of the Internet model with the application layer. The application layer describes all the high-level protocols that applications commonly interact with directly. Protocols at this layer handle how applications interface with the underlying transport layer to send and receive data.

To understand this layer, focus on the libraries or applications that implement the protocols underlying your application. For example, when a customer visits your website using a popular browser, the following steps happen:

1. The browser initiates library calls to obtain the IP address of the web server using the Domain Name System (DNS).

2. The browser initiates an HTTP request.

The DNS and HTTP protocols operate within the Internet model’s application layer.

Transport Layer

The next layer in the Internet model, the transport layer, handles the flow between hosts. Again, there are two primary protocols: the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP).

Historically, UDP has been the foundation of more rudimentary protocols, such as ping/ICMP, DHCP, ARP, and DNS, while TCP has been the foundation for more “interesting” protocols like HTTP, SSH, FTP, and SMB. However, this has been changing, as the qualities that make TCP more reliable on a per-session basis have performance bottlenecks in specific contexts.

UDP is a stateless protocol that makes a best-effort attempt to transfer messages but does not attempt to verify that network peers received messages; TCP, on the other hand, is a connection-oriented protocol that uses a three-way handshake to establish a reliable session with a network peer.

The essential characteristics of UDP are as follows:

Connectionless

UDP is not a session-oriented protocol. Network peers can exchange packets without first establishing a session.

Lossy

There is no support for error detection or correction. Applications must implement their fault-tolerance mechanisms.

Nonblocking

TCP is vulnerable to the “head of line blocking” problem, in which missing packets or nonsequential receipt of data can cause a session to get stuck and require retransmission from the point of the error. With UDP, nonsequential delivery is not a problem, and applications may selectively request retransmission of missing data without resending packets that have been successfully delivered.

By comparison, the essential characteristics of TCP are as follows:

Acknowledgment

The receiver notifies the sender of the data receipt for each packet. This receiver acknowledgment does not mean that the application has received or processed the data, only that it arrived at the intended destination.

Connection-oriented

The sender establishes a session before transmitting data.

Reliability

TCP keeps track of data that is sent and received. Receiver acknowledgments can be lost so that a receiver won’t acknowledge segments out of order; instead, it sends a duplicate of the last observed ordered packet or duplicate cumulative acknowledgment. This reliability can lengthen latency.

Flow control

The receiver notifies the sender of how much data can be received.

Notice that security wasn’t part of TCP or UDP’s design or fundamental characteristics. Instead, lack of security in the initial designs of these protocols drove additional complexity in application and system implementation and further changes in the protocols.

Network Layer

In the middle, the network layer translates between the transport and data link layers, enabling the delivery of packets based on a unique hierarchical address, the IP address.

The Internet Protocol (IP) is responsible for the rules for addressing and fragmentation of data between two systems. It handles the unique identification of network interfaces to deliver data packets using IP addresses. The IP breaks and reassembles packets as necessary when passing data through links with a smaller maximum transmission unit (MTU). IPv4 is the most widely deployed version of the IP with a 32-bit address space represented in a string of four binary octets or four decimal numbers separated by dots or quad-dotted decimal notation. The IPv6 standard brings advantages such as a 128-bit address space, more sophisticated routing capabilities, and better support for multicast addressing. However, IPv6 adoption has been slow partly because IPv4 and IPv6 are not interoperable, and making do with the shortcomings of IPv4 has generally been easier than porting everything to the new standard.

The underlying binary definition informs the range in decimal notation for IPv4 addresses of 0 to 255. In addition, RFCs define reserved ranges for private networks that aren’t routable via the public internet.

The network layer’s IP protocol is focused on providing a unique address for network peers to interact with, but it is not responsible for data link layer transmission, nor does it handle session management, which is dealt with at the transport layer.

Data Link Layer

Next, the data link layer uses the physical layer to send and receive packets.

The Address Resolution Protocol (ARP) handles hardware address discovery from a known IP address. Hardware addresses are also known as media access control (MAC) addresses. Each network interface controller (NIC) assigns it a unique MAC address.

The industry intended for MAC addresses to be globally unique, so network management devices and software assume this is true for device authentication and security. Duplicate MAC addresses appearing on the same network can cause problems. But duplicate MAC addresses do appear due to production errors in the manufacturing of hardware (or intentional software design with MAC randomization).⁴ Still, it can also happen with virtualized systems, such as VMs cloned from a reference image. However it happens, if multiple network hosts report having the same MAC address, network functionality errors and increased latency can occur.

Tip

People can mask the MAC address presented to the network through software. This is known as MAC spoofing. Some attackers use MAC spoofing as a layer 2 attack to attempt hijacking communication between two systems to hack into one of the systems.

The Reverse Address Resolution Protocol (RARP) examines IP address to hardware address mapping, which can help identify if multiple IP addresses are responding for a single MAC address. If you think you have a problem with two devices on the network sharing an IP address, perhaps because someone has assigned a static IP when a DHCP server has already allocated the same address to another host, RARP helps identify the culprit.

Physical Layer

The physical layer translates the binary data stream from upper layers into electrical, light, or radio signals that are transmitted across the underlying physical hardware. Each type of physical media has a different maximum length and speed.

Even when using cloud services, you still have to care about the physical layer, even though you don’t have to manage the racking and stacking of physical servers. Increased latency can be due to the physical routing between two points. For example, if something happens to a data center (lightning strikes or squirrels damage cables), resources might get redirected to alternate services farther away. Additionally, the networking gear within a data center may need to be rebooted or have a degraded cable or faulty network card. As a result, requests sent through those physical components may experience increased latency.

Wrapping Up

You’ll find a mix of these patterns (layered, microservices, and event-driven) and protocols in your environment. Understanding the system’s architecture informs how the components relate and communicate with one another, and your requirements impact the system’s reliability, maintainability, and scalability.

In the next chapter, I’ll share how to think about these patterns and interconnections and how they impact your choices for your computing environments within your organization.

¹ Martin Fowler, “Software Architecture Guide,” martinfowler.com, last modified August 1, 2019, www.martinfowler.com/architecture.

² Learn more about decomposing services from Chapter 3 of Sam Newman’s Building Microservices (O’Reilly).

³ CloudEvents is a community-driven effort to define a specification for describing event data in a standard way that can be implemented across different services and platforms.

⁴ Learn more about the issues with MAC randomization from “MAC Address Randomization: Privacy at the Cost of Security and Convenience”.