O'Reilly logo

Kubernetes Operators by Joshua Wood, Jason Dobies

Stay ahead with the world's most comprehensive technology and business learning platform.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, tutorials, and more.

Start Free Trial

No credit card required

Chapter 4. Operators in Go with the Operator SDK

While the Helm and Ansible Operators provide a way to rapidly create Operators, their functionality is ultimately limited by those underlying technologies. Advanced use cases, such as those that involve dynamically reacting to specific changes in the application or the cluster as a whole, require a more flexible solution.

The Go Operator provides that flexibility by allowing developers to use the Go programming language, including its ecosystem of external libraries, in their Operators.

As the process is slightly more involved than for the Helm or Go Operators, it makes sense to start with a summary of the high level steps:

  1. Initialize the Operator, creating the necessary code that will tie into Kubernetes and allow it to run the Operator as a controller.

  2. Create one or more custom resource definitions (CRDs) which model the application’s underlying business logic and provide the API for users to interact with.

  3. Create a controller for each CRD to handle the lifecycle of its resources.

  4. Build the Operator image and create the accompanying Kubernetes manifests to deploy the Operator and its RBAC components (Service Accounts, Roles, etc.).

While all of the above steps may be done manually, the Operator SDK provides commands that will automate much of the supporting code, allowing developers to focus on implementing the actual business logic of the Operator.

This chapter uses the Operator SDK to build the scaffold for implementing an Operator in Go (instructions on the SDK installation can be found in Chapter 4). We will explore the files that need to be edited with custom application logic and discuss some common practices for Operator development.

Initializing the Operator

Since the Operator is written in Go, the scaffold needs to be generated according to the language conventions. In particular, the Operator code must be located in the user’s GOPATH. See the GOPATH documentation for more information: https://github.com/golang/go/wiki/GOPATH

The SDK creates the necessary base files from which the operator is built:

$ OPERATOR_NAME=my-operator
$ operator-sdk new $OPERATOR_NAME --dep-manager=dep

The --dep-manager=dep flag tells the SDK to use the Go dependency management tool dep. Alternatively, the SDK can be configured to use Go modules instead. Keep in mind that module support must be activated before running the SDK. See the Go modules wiki for more information: https://github.com/golang/go/wiki/Modules

The SDK creates a new directory with the same name as $OPERATOR_NAME. As with the Adapter Operators described in chapter 6, this directory is a Git repository with all of the generated files already committed. Optionally, the git remote command can be used to add an external repository, such as GitHub, and push the initial commit.

The SDK produces hundreds of files, both generated and vendor files, that are used by the Operator. Conveniently, nearly all of them can be ignored and do not need to be manually edited. The files necessary to fulfill custom logic for an Operator are generated in the sections later in this chapter.

Operator Scope

One of the first decisions you need to make is how the Operator will be scoped. There are two options:

  • Namespaced, which limits the Operator to managing resources in a single namespace

  • Cluster, which allows the Operator to manage resources across the entire cluster

By default, Operators generated by the SDK are namespace-scoped.

While namespace-scoped Operators are often preferable, changing an SDK generated Operator to be cluster-scoped is possible. The following changes must be made to enable the Operator to work at the cluster level:

  • deploy/operator.yaml

    • Change the value of the WATCH_NAMESPACE variable to "", indicating all namespaces should be watched instead of only the namespace in which the Operator pod is deployed.

  • deploy/role.yaml

    • Change the kind from Role to ClusterRole to enable permissions outside of the Operator pod’s namespace.

  • deploy/role_binding.yaml

    • Change the kind from RoleBinding to ClusterRoleBinding.

    • Under roleRef, change the kind to ClusterRole.

    • Under subjects, add the key namespace with the value being the namespace in which the Operator pod will be deployed.

Additionally, the generated custom resource definitions (see Custom Resource Definitions below) need to be updated to indicate that the definition is cluster-scoped. This change is made in two places:

  • Under the spec section of the CRD file, the scope field must be changed to Cluster instead of the default value of Namespaced.

  • In the _types.go file for the CRD, the tag // +genclient:nonNamespaced must be added above the struct for the custom resource (this will be the same name as the kind field used to create it). This ensures that future calls to the Operator SDK to refresh the CRD will not reset the value to the default.

For example, the following modifications to the VisitorsApp struct indicate it is cluster-scoped:

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

// VisitorsApp is the Schema for the visitorsapps API
// +k8s:openapi-gen=true
// +kubebuilder:subresource:status
// +genclient:nonNamespaced  1
type VisitorsApp struct {
1

The tag is added before the resource type struct.

Custom Resource Definitions

In chapter 6, we discussed the role of custom resource definitions when creating an Operator. New custom resource definitions are added to the Operator using the add api command. The example below generates the CRD for the entire Visitors Site example used in this book (using the arbitrary “example.com” for demonstration purposes):

$ operator-sdk add api --api-version=example.com/v1 --kind=VisitorsApp

This command generates a number of files. In the files below, note how both the api-version and custom resource type name (kind) are used in the generated names (file paths are relative to the Operator project root).

  • deploy/crds/example_v1_visitorsapp-cr.yaml

    • This is an example custom resource of the generated type. It is pre-populated with the appropriate api-version and kind, as well as an example name. The spec section must be filled out with values relevant to the created CRD.

  • deploy/crds/example_v1_visitorsapp_crd.yaml

    • This file is the beginnings of a custom resource definition template. Many fields related to the name of the type (such as plural and list variations) are generated and included. More detail on flushing this out is included later in this section.

  • pkg/apis/example/v1/visitorsapp_types.go

    • This file contains a number of struct objects that are used throughout the Operator codebase. This file, unlike many of the generated Go files, is intended to be edited.

The add api command builds the appropriate scaffold code, but before the resource type can be used, the set of configuration values specified when creating a new resource must be defined. These are defined both in the definition template itself as well as the Go objects.

Define the Go Types

In the *_types.go file (in this example, visitorsapp_types.go), there are two struct objects that need to be addressed:

  • The spec object (in this example, VisitorsAppSpec) must include all possible configuration values that may be specified for resources of this type. Each configuration value is defined using the following:

    • The name of the variable as it will be referenced from within the Operator code (following Go conventions and beginning with a capital letter for language visibility purposes)

    • The Go type for the variable

    • The name of the field as it will be specified in the custom resource itself (in other words, the JSON or YAML template used to create the resource)

  • The status object (in this example, VisitorsAppStatus) must include all possible values that may be set by the Operator to convey the state of the custom resource. Each value is defined using the following:

    • The name of the variable as it will be referenced from within the Operator code (following Go conventions and beginning with a capital letter for visibility purposes)

    • The Go type for the variable

    • The name of the field as it will appear in the description of the custom resource itself (for example, when getting the resource with the -o yaml flag)

In the VisitorsApp example, the following values are specified for each deployment:

  • Size - The number of backend replicas to create

  • Title - The text to be displayed on the frontend web page

It is important to realize that despite the fact that these values are used in different pods in the application, they are all included in the single custom resource definition. From the user’s perspective, they are attributes of the overall application. The details of how those values are used is the responsibility of the Operator.

The VisitorsApp uses the following values in the status of each resource:

  • Backend Image - Indicates the image and version used to deploy the backend pods

  • Frontend Image - Indicates the image and version used to deploy the frontend pod

These changes are reflected in the following snippet from the visitorsapp_types.go file:

type VisitorsAppSpec struct {
    Size       int32  `json:"size"`
    Title      string `json:"title"`
}

type VisitorsAppStatus struct {
    BackendImage  string `json:"backendImage"`
    FrontendImage string `json:"frontendImage"`
}

The rest of the visitorsapp_types.go file does not require any further changes.

After any change to a *_types.go file, the SDK is used to update any generated code that manipulates these objects through the generate command (this should be run in the root directory of the Operator):

$ operator-sdk generate k8s

Define the CRD Validation

The additions to the types file are useful to the Operator code, but provide no insight to the end user creating the resource. Those additions are made to the custom resource definition itself.

Similar to the types file, the additions to the CRD are made in the spec and status sections.

There are two primary edits done to the spec section of the CRD:

  • A properties map is added. An entry should be added for each of the attributes that may be specified for custom resources of this type, along with information on the parameter’s type and allowed values.

  • Optionally, a list named required can be added to have Kubernetes enforce the presence of certain spec properties. The name of each required property should be added as an entry in this list. If any of these properties are omitted during resource creation, Kubernetes will reject the resource.

The status section is also flushed out with property information following the same conventions as for spec, however there is no need to add a required field.

In both cases, the existing line type: object remains; the new additions are added at the same level as this type declaration. Both the spec and status fields can be found in the following section of the CRD:

spec → validation → openAPIV3Schema → properties

For the VisitorsApp example, the additions are as follows:

spec:
    type: object
    properties:
        size:
            type: integer
        title:
            type: string
    required:
    - size
status:
    type: object
    properties:
        backendImage:
            type: string
        frontendImage:
            type: string

More information on defining custom resource definitions can be found in the Kubernetes documentation: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/

Controller

The CRD, and its associated types file in Go, define the inbound API through which users will communicate. Inside the Operator, a controller is needed to watch for changes to the custom resource and react accordingly.

Similar to adding a CRD, the controller code is generated using the SDK. Continuing with the Visitors Site example, the api-version and kind of the previously generated resource definition are specified to scope the controller to that type:

$ operator-sdk add controller --api-version=example.com/v1 --kind=VisitorsApp

As with the custom resource definition, a number of files are generated by this command. Of particular interest is the controller file, which is located and named according to the associated kind; the other generated files do not need to be manually edited. For example, the command above creates:

pkg/controller/visitorsapp/visitorsapp_controller.go

The controller is responsible for “reconciling” a specific resource. The notion of a single reconcile operation is consistent with the declarative model that Kubernetes follows. Instead of having explicit handling for events such as add, delete, or update, the controller is passed the current state of the resource. It is up to the controller to determine what changes need to be made to update reality to reflect the state described in the resource.

While the bulk of the Operator logic will reside in the Reconcile function, there is one other location of interest in the *_controller.go file. The add function is used to establish the watches that will be used to trigger reconcile events. Two such watches are created in the code generated by the SDK.

The first watch is established to listen for changes to the primary resource that the controller monitors. This defaults to watching for resources of the same type as the “kind” parameter used when first generating the controller. In most cases, this does not need to be changed.

The second watch, or more accurately, series of watches, are used to listen for changes to any child resources created to support the primary resource. For example, creating a VisitorsApp resource will result in the creation of multiple deployment and service objects to support its function. A watch should be added for each of these child types, being careful to scope the watch to only child resources whose owner is of the same type as the primary resource. For example:

err = c.Watch(&source.Kind{Type: &appsv1.Deployment{}}, &handler.EnqueueRequestForOwner{
    IsController: true,
    OwnerType:    &examplev1.VisitorsApp{},
})
if err != nil {
    return err
}

err = c.Watch(&source.Kind{Type: &corev1.Service{}}, &handler.EnqueueRequestForOwner{
    IsController: true,
    OwnerType:    &examplev1.VisitorsApp{},
})
if err != nil {
    return err
}

For the watches created in the snippet above, there are two areas of interest:

  • The value for Type in the constructor indicates the child resource type that is being watched. A separate watch should be created for each child resource type.

  • The value for OwnerType is set to the primary resource type, scoping the watch and indicating that a reconcile should be triggered for the parent resource.

The Reconcile Function

The Reconcile function, also known as the reconcile loop, is where the Operator’s logic resides. The purpose of this function is to resolve the actual state against the state requested by the resource.

Important

As the reconcile function will be invoked multiple times throughout the lifecycle of a resource, it is important that the implementation be idempotent to prevent the creation of duplicate child resources.

The Reconcile function returns two objects: a ReconcileResult instance and an error (if one was encountered). Those return values indicate whether or not the request should be requeued. In other words, the Operator tells Kubernetes if the reconcile loop should execute again. The following describes the possible outcomes based on the return values:

  • return reconcile.Result{}, nil

    • Indicates the reconcile process finished with no errors and does not require another pass through the reconcile loop.

  • return reconcile.Result{}, err

    • The reconcile failed due to an error and should be requeued to try again.

  • return reconcile.Result{Requeue: true}, nil

    • The reconcile did not encounter an error, but it should be requeued to run for another iteration.

  • return reconcile.Result{RequeueAfter: time.Second*5}, nil

    • Similar to the previous result, this will wait for the specified amount of time before requeuing the request.

Operator Writing Tips

It is impossible to cover all of the conceivable uses of Operators in a single book. The differences in application installation and upgrade alone are too many to enumerate, and those represent only the first two layers of the Operator Maturity Model. Instead, we will cover some general guidelines to get you started with the basic handling and manipulation of resources in Kubernetes.

Since the Go Operator makes heavy use of the Go Kubernetes libraries, it may be useful to review the API documentation found at https://godoc.org/k8s.io/api. In particular, the core/v1 and apps/v1 modules is frequently used to interact with the commonly used Kubernetes resources.

Parent Resource

The first step commonly performed in the Reconcile method is to retrieve the primary resource that triggered the reconcile request. The Operator SDK generates the code for this, which should look similar to the following that was generated in to the Visitors Site example:

// Fetch the VisitorsApp instance
instance := &examplev1.VisitorsApp{}
err := r.client.Get(context.TODO(), request.NamespacedName, instance) 12

if err != nil {
    if errors.IsNotFound(err) {
        // Request object not found, could have been deleted after reconcile request.
        // Owned objects are automatically garbage collected. For
           additional cleanup logic use finalizers.
        // Return and don't requeue
        return reconcile.Result{}, nil 3
    }
    // Error reading the object - requeue the request.
    return reconcile.Result{}, err
}
1

Populates the previously created VisitorsApp object with the values from the resource that triggered the reconcile.

2

The variable r is the reconciler object that the Reconcile method is called on. It provides the client object which is an authenticated client into the Kubernetes API.

3

Reconcile is called when a resource is deleted, in which case the Get call returns an error. More information on handling deleted resources can be found later in this section.

The retrieved instance is primarily used for two purposes:

  • Retrieving configuration values about the resource from its Spec field.

  • Setting the current state of the resource, using its Status field, and saving that updated information into Kubernetes.

Child Resource Creation

One of the first tasks typically implemented in an Operator is to deploy the resources necessary to get the application running. It is critical that this operation be idempotent; subsequent calls to the Reconcile method should ensure the resource is running rather than creating duplicate resources.

These child resources commonly include, but are not limited to, deployment and service objects. The handling for them is similar and straightforward: check to see if the resource is present in the namespace and, if it is not, create it.

The example snippet below checks for the existence of a Deployment in the target namespace:

found := &appsv1.Deployment{}
findMe := types.NamespacedName{
    Name:      "myDeployment",  1
    Namespace: instance.Namespace,  2
}
err := r.client.Get(context.TODO(), findMe, found)
if err != nil && errors.IsNotFound(err) {
    // Creation logic 3
}
1

The Operator knows the names, or at least how to derive them, of the child resources it created. In real use cases, "myDeployment" will be replaced with the same name used by the Operator when the deployment was created, taking care to ensure uniqueness relative to the namespace as appropriate.

2

The instance variable was set in the previous snippet and refers to the object representing the primary resource being reconciled.

3

At this point, the child resource was not found and no further errors were retrieved from the Kubernetes API, so the resource creation logic should be executed.

Resources are created by populating the necessary Kubernetes objects and using the client to request they be created. Consult the Kubernetes Go client API for specifications on how to instantiate the resource for each type. Many of the desired specs can be found in either the core/v1 or apps/v1 modules.

As an example, the snippet below is used to create a Deployment specification for the MySQL database used in the Visitors Site example application:

labels := map[string]string {
    "app":             "visitors",
    "visitorssite_cr": instance.Name,
    "tier":            "mysql",
}
size := int32(1)  1

dep := &appsv1.Deployment{
    ObjectMeta: metav1.ObjectMeta{
        Name:        "mysql-backend-service", 2
        Namespace:     instance.Namespace,
    },
    Spec: appsv1.DeploymentSpec{
        Replicas: &size,
        Selector: &metav1.LabelSelector{
            MatchLabels: labels,
        },
        Template: corev1.PodTemplateSpec{
            ObjectMeta: metav1.ObjectMeta{
                Labels: labels,
            },
            Spec: corev1.PodSpec{
                Containers: []corev1.Container{{
                    Image:    "mysql:5.7",
                    Name:    "visitors-mysql",
                    Ports:    []corev1.ContainerPort{{
                        ContainerPort:     3306,
                        Name:            "mysql",
                    }},
                    Env:    []corev1.EnvVar{ 3
                        {
                            Name:    "MYSQL_ROOT_PASSWORD",
                            Value:     "password",
                        },
                        {
                            Name:    "MYSQL_DATABASE",
                            Value:    "visitors",
                        },
                        {
                            Name:    "MYSQL_USER",
                            Value:    "visitors",
                        },
                        {
                            Name:    "MYSQL_PASSWORD",
                            Value:    "visitors",
                        },
                    },
                }},
            },
        },
    },
}

controllerutil.SetControllerReference(instance, dep, r.scheme) 4
1

In many cases, the number of deployed pods is read from the primary resource’s spec. For simplicity, this is hardcoded to 1 in this example.

2

This is the value used in the snippet above when attempting to see if the Deployment exists (in that example, it was set to “myDeployment”).

3

For this example, these are hardcoded values. Care should be taken to generate randomized values as appropriate.

4

This is, arguably, the most important line in the definition. This establishes the parent/child relationship between the primary resource (VisitorsApp) and the child (Deployment). This relationship is used by Kubernetes for certain operations (see Child Resource Deletion later in this section).

The structure of the Go representation of the deployment closely resembles the YAML definition. Again, consult the API documentation for the specifics on how to use the Go object models.

Regardless of the child resource type (Deployment, Service, etc), it is created using the client:

createMe := // Deployment instance from above

// Create the service
err = r.client.Create(context.TODO(), createMe)

if err != nil {
    // Creation failed
    return &reconcile.Result{}, err
} else {
    // Creation was successful
    return nil, nil
}

Child Resource Deletion

In most cases, deleting child resources is significantly simpler than creating them: Kubernetes will do it for you. If the child resource’s owner type is correctly set to the primary resource, when the parent is deleted, Kubernetes garbage collection will automatically clean up all of its child resources.

It is important to understand that the Reconcile function is still called when a resource is deleted. Kubernetes garbage collection still takes place and the primary resource will not be found. See the Parent Resource section above for an example of the code that checks for this situation.

There are times, however, where specific cleanup logic is required. The approach in such instances is to block the deletion of the primary resource through the use of a finalizer.

A finalizer is simply a series of strings on a resource. If one or more finalizers are present on a resource, the metadata.deletionTimestamp field of the object will be populated, signifying the desire to delete the resource. However, Kubernetes will only perform the actual delete once all of the finalizers have been removed.

Using this construct, developers can block the automatic deletion of a resource until the Operator has a chance to perform its own cleanup step. Once the Operator has finished with the necessary cleanup, it removes the finalizer, unblocking Kubernetes from performing its normal deletion steps.

The following snippet demonstrates using a finalizer to provide a window in which the Operator can take pre-deletion steps. This code will execute after the retrieval of the instance object as outlined in the Parent Resource section:

finalizer := "visitors.example.com"

beingDeleted := instance.GetDeletionTimestamp() != nil  1
if beingDeleted {
    if contains(instance.GetFinalizers(), finalizer) {

        // Perform finalization logic. If this fails, leave the finalizer
        // intact and requeue the reconcile request to attempt the clean
        // up again without allowing Kubernetes to actually delete
        // the resource.

        instance.SetFinalizers(remove(instance.GetFinalizers(), finalizer)) 2
        err := r.client.Update(context.TODO(), instance)
        if err != nil {
            return reconcile.Result{}, err
        }
    }
    return reconcile.Result{}, nil
}
1

The presence of a deletion timestamp indicates that a delete was requested but is being blocked by one or more finalizers.

2

Once the clean up tasks have finished, the finalizer is removed so Kubernetes can continue with the resource clean up.

Run the Operator

The Operator SDK provides a means of running an Operator outside of a running cluster. This helps speed development and testing time by removing the need to go through the image creation and hosting steps. The process running the Operator may be outside of the cluster, but Kubernetes will treat it as it does any other controller.

The high level steps for testing an Operator are as follows:

  1. Deploy the custom resource definition. This only needs to be performed once, unless there are further changes to the CRD. In those cases, the kubectl apply command should be run again to apply any changes.

$ cd deploy/crds
$ kubectl apply -f *_crd.yaml
  1. Start the operator in local mode. The Operator SDK uses credentials from the kubectl configuration file to connect to the cluster and attach the Operator. The running process acts as if it was an Operator pod running inside of the cluster, with logging information being written to standard out.

$ export OPERATOR_NAME=<operator-name>
$ operator-sdk up local --namespace default

The --namespace flag indicates the namespace in which the Operator will appear to be running.

  1. Deploy an example resource. An example custom resource is generated along with the CRD. It is located in the same directory and be named similarly to the CRD, ending in _cr.yaml instead to denote its difference.

In most cases, the spec section of this file must be edited to provide the necessary configuration values. The custom resource is then deployed using kubectl:

$ kubectl apply -f *_cr.yaml
  1. Stop the running Operator process. The Operator process is stopped by pressing ctrl+c. Unless finalizers are used, this is safe to do before deleting the custom resource itself, as Kubernetes will use the parent/child relationship of its resources to cleanup any dependent objects.

Note

The process above is useful for development purposes, but for production, Operators are delivered as images. See Appendix 1 for more information on how to build and deploy an Operator as a container inside of the cluster.

Visitors Site Example

The full codebase for the Visitors Site Operator is too large to include in this book. For reference, the source for the fully built Operator is on GitHub:

https://github.com/kubernetes-operators-book/visitors-operator

Many of the files in that repository were generated by the Operator SDK. The following files contain the changes made to run the Visitors site:

  • deploy/crds/*

    • This directory contains the custom resource definition and an example custom resource. The custom resource has been populated with example data.

  • pkg/apis/example/v1/visitorsapp_types.go

    • Go objects that represent the custom resource, including its spec and status fields.

  • pkg/controller/visitorsapp/

    • backend.go, frontend.go, mysql.go

      • These files contain all of the information specific to deploying those components of the Visitors Site. This includes the deployments and services that are maintained by the Operator, as well as the logic to handle updating existing resources in the event a relevant change is made to the custom resource spec.

    • common.go

      • This file contains utility methods used to ensure the deployments and services are running, creating them if necessary.

    • visitorsapp_controller.go

      • This was initially generated by the Operator SDK and was modified for the Visitors Site specific logic. Many of the changes were made to the Reconcile method, which drives the overall flow of the Operator by calling out to functions in the previously described files.

Summary

Writing an Operator requires a considerable amount of code to tie into Kubernetes as a controller. The Operator SDK eases development by generating much of this boilerplate code, letting developers focus on the business logic aspects. The SDK also provides utilities for building and testing Operators, greatly reducing the effort needed to go from inception to a running Operator.

With Safari, you learn the way you learn best. Get unlimited access to videos, live online training, learning paths, books, interactive tutorials, and more.

Start Free Trial

No credit card required