97 Things Every Engineering Manager Should Know

Chapter 73. Reconciliation Loops

Kevin Stewart

In distributed systems, there has been a continual push to make systems self-healing. We can find one implementation of this in Kubernetes. Kubernetes is based on the concept of a declarative specification of the desired state of the cluster and the use of reconciliation loops to drive the actual state toward the desired state.

In a Kubernetes cluster, there are several components that work together to implement this behavior. The primary component is the pod, which has one or more containers, and is the unit that gets scheduled onto a node in the cluster. Containers reside in an image registry, and when placed on a node are governed by the container runtime on that node and a process known as a kubelet.

One node in the cluster is considered the master node where all the control-plane components reside. The API server handles incoming requests and updates the etcd database with state changes. The scheduler is responsible for determining which nodes are appropriate to allocate a given workload. Finally, the controller manager coordinates a set of controllers that make up the reconciliation loops that drive the entire cluster to the desired state.

In a properly managed Kubernetes cluster, this system works fairly well. There is mostly a clear separation of duties and loose coupling. Although there is some initial pain in getting ...