book

Core Kubernetes

by Jay Vyas, Christopher Love

June 2022

Intermediate to advanced

336 pages

10h 6m

English

Manning Publications

Read now

Unlock full access

prefaceacknowledgmentsabout this bookWho should read this bookHow this book is organized: A road mapAbout the codeliveBook discussion forumabout the authorsabout the cover illustration
1.1 Reviewing a few key terms before we get started1.2 The infrastructure drift problem and Kubernetes1.3 Containers and images1.4 Core foundation of Kubernetes1.4.1 All infrastructure rules in Kubernetes are managed as plain YAML1.5 Kubernetes features1.6 Kubernetes components and architecture1.6.1 The Kubernetes API1.6.2 Example one: An online retailer1.6.3 Example two: An online giving solution1.7 When not to use KubernetesSummary
2.1 An example web application2.1.1 Infrastructure for our web application2.1.2 Operational requirements2.2 What is a Pod?2.2.1 A bunch of Linux namespaces2.2.2 Kubernetes, infrastructure, and the Pod2.2.3 The Node API object2.2.4 Our web application and the control plane2.3 Creating a web application with kubectl2.3.1 The Kubernetes API server: kube-apiserver2.3.2 The Kubernetes scheduler: kube-scheduler2.3.3 Infrastructure controllers2.4 Scaling, highly available applications, and the control plane2.4.1 Autoscaling2.4.2 Cost managementSummary
3.1 Looking at Kubernetes primitives with kind3.2 What is a Linux primitive?3.2.1 Linux primitives are resource management tools3.2.2 Everything is a file (or a file descriptor)3.2.3 Files are composable3.2.4 Setting up kind3.3 Using Linux primitives in Kubernetes3.3.1 The prerequisites for running a Pod3.3.2 Running a simple Pod3.3.3 Exploring the Pod’s Linux dependencies3.4 Building a Pod from scratch3.4.1 Creating an isolated process with chroot3.4.2 Using mount to give our process data to work with3.4.3 Securing our process with unshare3.4.4 Creating a network namespace3.4.5 Checking whether a process is healthy3.4.6 Adjusting CPU with cgroups3.4.7 Creating a resources stanza3.5 Using our Pod in the real world3.5.1 The networking problem3.5.2 Utilizing iptables to understand how kube-proxy implements Kubernetes services3.5.3 Using the kube-dns Pod3.5.4 Considering other issuesSummary
4.1 Pods are idle until the prep work completes4.2 Processes and threads in Linux4.2.1 systemd and the init process4.2.2 cgroups for our process4.2.3 Implementing cgroups for a normal Pod4.3 Testing the cgroups4.4 How the kubelet manages cgroups4.5 Diving into how the kubelet manages resources4.5.1 Why can’t the OS use swap in Kubernetes?4.5.2 Hack: The poor man’s priority knob4.5.3 Hack: Editing HugePages with init containers4.5.4 QoS classes: Why they matter and how they work4.5.5 Creating QoS classes by setting resources4.6 Monitoring the Linux kernel with Prometheus, cAdvisor, and the API server4.6.1 Metrics are cheap to publish and extremely valuable4.6.2 Why do I need Prometheus?4.6.3 Creating a local Prometheus monitoring service4.6.4 Characterizing an outage in PrometheusSummary

5.1 Why we need software-defined networks in Kubernetes5.2 Implementing the service side of the Kubernetes SDN: The kube-proxy5.2.1 The kube-proxy’s data plane5.2.2 What about NodePorts?5.3 CNI providers5.4 Diving into two CNI networking plugins: Calico and Antrea5.4.1 The architecture of a CNI plugin5.4.2 Let’s play with some CNIs5.4.3 Installing the Calico CNI provider5.4.4 Kubernetes networking with OVS and Antrea5.4.5 A note on CNI providers and kube-proxy on different OSsSummary
6.1 Sonobuoy: A tool for confirming your cluster is functioning6.1.1 Tracing data paths for Pods in a real cluster6.1.2 Setting up a cluster with the Antrea CNI provider6.2 Inspecting CNI routing on different providers with the arp and ip commands6.2.1 What is an IP tunnel and why do CNI providers use them?6.2.2 How many packets are flowing through the network interfaces for our CNI?6.2.3 Routes6.2.4 CNI-specific tooling: Open vSwitch (OVS)6.2.5 Tracing the data path of active containers with tcpdump6.3 The kube-proxy and iptables6.3.1 iptables-save and the diff tool6.3.2 Looking at how network policies modify CNI rules6.3.3 How are these policies implemented?6.4 Ingress controllers6.4.1 Setting up Contour and kind to explore ingress controllers6.4.2 Setting up a simple web server PodSummary
7.1 A quick detour: The virtual filesystem (VFS) in Linux7.2 Three types of storage requirements for Kubernetes7.3 Let’s create a PVC in our kind cluster7.4 The container storage interface (CSI)7.4.1 The in-tree provider problem7.4.2 CSI as a specification that works inside of Kubernetes7.4.3 CSI: How a storage driver works7.4.4 Bind mounting7.5 A quick look at a few running CSI drivers7.5.1 The controller7.5.2 The node interface7.5.3 CSI on non-Linux OSsSummary
8.1 A microcosm of the broader Kubernetes ecosystem: Dynamic storage8.1.1 Managing storage on the fly: Dynamic provisioning8.1.2 Local storage compared with emptyDir8.1.3 PersistentVolumes8.1.4 CSI (container storage interface)8.2 Dynamic provisioning benefits from CSI but is orthogonal8.2.1 StorageClasses8.2.2 Back to the data center stuff8.3 Kubernetes use cases for storage8.3.1 Secrets: Sharing files ephemerally8.4 What does a dynamic storage provider typically look like?8.5 hostPath for system control and/or data access8.5.1 hostPaths, CSI, and CNI: A canonical use case8.5.2 Cassandra: An example of real-world Kubernetes application storage8.5.3 Advanced storage functionality and the Kubernetes storage model8.6 Further readingSummary
9.1 The kubelet and the node9.2 The core kubelet9.2.1 Container runtimes: Standards and conventions9.2.2 The kubelet configurations and its API9.3 Creating a Pod and seeing it in action9.3.1 Starting the kubelet binary9.3.2 After startup: Node life cycle9.3.3 Leasing and locking in etcd and the evolution of the node lease9.3.4 The kubelet’s management of the Pod life cycle9.3.5 CRI, containers, and images: How they are related9.3.6 The kubelet doesn’t run containers: That’s the CRI’s job9.3.7 Pause container: An “aha” moment9.4 The Container Runtime Interface (CRI)9.4.1 Telling Kubernetes where our container runtime lives9.4.2 The CRI routines9.4.3 The kubelet’s abstraction around CRI: The GenericRuntimeManager9.4.4 How is the CRI invoked?9.5 The kubelet’s interfaces9.5.1 The Runtime internal interface9.5.2 How the kubelet pulls images: The ImageService interface9.5.3 Giving ImagePullSecrets to the kubelet9.6 Further readingSummary
10.1 A brief intro to DNS (and CoreDNS)10.1.1 NXDOMAINs, A records, and CNAME records10.1.2 Pods need internal DNS10.2 Why StatefulSets instead of Deployments?10.2.1 DNS with headless services10.2.2 Persistent DNS records in StatefulSets10.2.3 Using a polyglot deployment to explore Pod DNS properties10.3 The resolv.conf file10.3.1 A quick note about routing10.3.2 CoreDNS: The upstream resolver for the ClusterFirst Pod DNS10.3.3 Hacking the CoreDNS plugin configurationSummary
11.1 Investigating the control plane11.2 API server details11.2.1 API objects and custom API objects11.2.2 Custom resource definitions (CRDs)11.2.3 Scheduler details11.2.4 Recap of scheduling11.3 The controller manager11.3.1 Storage11.3.2 Service accounts and tokens11.4 Kubernetes cloud controller managers (CCMs)11.5 Further readingSummary
12.1 Notes for the impatient12.1.1 Visualizing etcd performance with Prometheus12.1.2 Knowing when to tune etcd12.1.3 Example: A quick health check of etcd12.1.4 etcd v3 vs. v212.2 etcd as a data store12.2.1 The watch: Can you run Kubernetes on other databases?12.2.2 Strict consistency12.2.3 fsync operations make etcd consistent12.3 Looking at the interface for Kubernetes to etcd12.4 etcd’s job is to keep the facts straight12.4.1 The etcd write-ahead log12.4.2 Effect on Kubernetes12.5 The CAP theorem12.6 Load balancing at the client level and etcd12.6.1 Size limitations: What (not) to worry about12.7 etcd encryption at rest12.8 Performance and fault tolerance of etcd at a global scale12.9 Heartbeat times for a highly distributed etcd12.10 Setting an etcd client up on a kind cluster12.10.1 Running etcd in non-Linux environmentsSummary
13.1 Blast radius13.1.1 Vulnerabilities13.1.2 Intrusion13.2 Container security13.2.1 Plan to update containers and custom software13.2.2 Container screening13.2.3 Container users—do not run as root13.2.4 Use the smallest container13.2.5 Container provenance13.2.6 Linters for containers13.3 Pod security13.3.1 Security context13.3.2 Escalated permissions and capabilities13.3.3 Pod Security Policies (PSPs)13.3.4 Do not automount the service account token13.3.5 Root-like Pods13.3.6 The security outskirtsSummary
14.1 Node security14.1.1 TLS certificates14.1.2 Immutable OSs vs. patching nodes14.1.3 Isolated container runtimes14.1.4 Resource attacks14.1.5 CPU units14.1.6 Memory units14.1.7 Storage units14.1.8 Host networks vs. Pod networks14.1.9 Pod example14.2 API server security14.2.1 Role-based access control (RBAC)14.2.2 RBAC API definition14.2.3 Resources and subresources14.2.4 Subjects and RBAC14.2.5 Debugging RBAC14.3 Authn, Authz, and Secrets14.3.1 IAM service accounts: Securing your cloud APIs14.3.2 Access to cloud resources14.3.3 Private API servers14.4 Network security14.4.1 Network policies14.4.2 Load balancers14.4.3 Open Policy Agent (OPA)14.4.4 Multi-tenancy14.5 Kubernetes tipsSummary
15.1 Thinking about apps in Kubernetes15.1.1 Application scope influences what tools you should use15.2 Microservice apps typically require thousands of lines of configuration code15.3 Rethinking our Guestbook app installation for the real world15.4 Installing the Carvel toolkit15.4.1 Part 1: Modularizing our resources into separate files15.4.2 Part 2: Patching our application files with ytt15.4.3 Part 3: Managing and deploying Guestbook as a single application15.4.4 Part 4: Constructing a kapp Operator to package and manage our application15.5 Revisiting the Kubernetes Operator15.6 Tanzu Community Edition: An end-to-end example of the Carvel toolkitSummary

Content preview from Core Kubernetes

6 Troubleshooting large-scale network errors

This chapter covers

Confirming cluster functionality with Sonobuoy
Tracing a Pod’s data path
Using the arp and ip commands to inspect CNI routing
A deeper look at kube-proxy and iptables
An introduction to Layer 7 networking (the ingress resource)

In this chapter, we’ll go over a few touchpoints for troubleshooting large-scale network errors. We also introduce Sonobuoy, a Swiss Army knife for certifying, diagnosing, and testing the functionality of live Kubernetes clusters, which is a commonly used diagnosis tool for Kubernetes.