Chapter 4. Configuration, Secrets, and RBAC
The composable nature of containers allows us as operators to introduce configuration data into a container at runtime. This makes it possible for us to decouple an application’s function from the environment it runs in. By means of the conventions allowed in the container runtime to pass through either environment variables or mount external volumes into a container at runtime, you can effectively change the configuration of the application upon its instantiation. As a developer, it is important to take into consideration the dynamic nature of this behavior and allow for the use of environment variables or the reading of configuration data from a specific path available to the application runtime user.
When moving sensitive data such as secrets into a native Kubernetes API object, it is important to understand how Kubernetes secures access to the API. The most commonly implemented security method in use in Kubernetes is Role-Based Access Control (RBAC) to implement a fine-grained permission structure around actions that can be taken against the API by specific users or groups. This chapter covers some of the best practices regarding RBAC and also provides a small primer.
Configuration Through ConfigMaps and Secrets
Kubernetes allows you to natively provide configuration information to our applications through ConfigMaps or secret resources. The main differentiator between the two is the way a pod stores the receiving information and how the data is stored in the etcd data store.
ConfigMaps
It is very common to have applications consume configuration information through some type of mechanism such as command-line arguments, environment variables, or files that are available to the system. Containers allow the developer to decouple this configuration information from the application, which allows for true application portability. The ConfigMap API allows for the injection of supplied configuration information. ConfigMaps are very adaptable to the application’s requirements and can provide key/value pairs or complex bulk data such as JSON, XML, or proprietary configuration data.
The ConfigMaps not only provide configuration information for pods, but can also provide information to be consumed for more complex system services such as controllers, CRDs, operators, and so on. As mentioned earlier, the ConfigMap API is meant more for string data that is not really sensitive data. If your application requires more sensitive data, the Secrets API is more appropriate.
For your application to use the ConfigMap data, it can be injected as either a volume mounted into the pod or as environment variables.
Secrets
Many of the attributes and reasons for which you would want to use a ConfigMap apply to secrets. The main differences lie in the fundamental nature of a Secret. Secret data should be stored and handled in a way that can be easily hidden and possibly encrypted at rest if the environment is configured as such. The Secret data is represented as base64-encoded information, and it is critical to understand that this is not encrypted. As soon as the secret is injected into the pod, the pod itself can see the secret data in plain text.
Secret data is meant to be small amounts of data, limited by default in Kubernetes to 1 MB in size, for the base64-encoded data, so ensure that the actual data is approximately 750 KB because of the overhead of the encoding. There are three types of secrets in Kubernetes:
generic
-
This is typically just regular key/value pairs that are created from a file, a directory, or from string literals using the
--from-literal=
parameter, as follows:kubectl create secret generic mysecret --from-literal
=
key1
=
$3cr3t1
--from-literal=
key2
=
@3cr3t2`
docker-registry
-
This is used by the kubelet when passed in a pod template if there is an
imagePullsecret
to provide the credentials needed to authenticate to a private Docker registry:kubectl create secret docker-registry registryKey --docker-server myreg.azurecr.io --docker-username myreg --docker-password
$up3r$3cr3tP
@ssw0rd --docker-email ignore@dummy.com tls
-
This creates a Transport Layer Security (TLS) secret from a valid public/private key pair. As long as the cert is in a valid PEM format, the key pair will be encoded as a secret and can be passed to the pod to use for SSL/TLS needs:
kubectl create secret tls www-tls --key
=
./path_to_key/wwwtls.key --cert=
./path_to_crt/wwwtls.crt
Secrets are also mounted into tmpfs only on the nodes that have a pod that requires the secret and are deleted when the pod that needs it is gone. This prevents any secrets from being left behind on the disk of the node. Although this might seem secure, it is important to know that by default, secrets are stored in the etcd datastore of Kubernetes in plain text, and it is important that the system administrators or cloud service provider take efforts to ensure that the security of the etcd environment, including mTLS between the etcd nodes and enabling encryption at rest for the etcd data. More recent versions of Kubernetes use etcd3 and have the ability to enable etcd native encryption; however, this is a manual process that must be configured in the API server configuration by specifying a provider and the proper key media to properly encrypt secret data held in etcd. As of Kubernetes v1.10 (it has been promoted to beta in v1.12), we have the KMS provider, which promises to provide a more secure key process by using third-party KMS systems to hold the proper keys.
Common Best Practices for the ConfigMap and Secrets APIs
The majority of issues that arise from the use of a ConfigMap or secret are incorrect assumptions on how changes are handled when the data held by the object is updated. By understanding the rules of the road and adding a few tricks to make it easier to abide by those rules, you can steer away from trouble:
-
To support dynamic changes to your application without having to redeploy new versions of the pods, mount your ConfigMaps/Secrets as a volume and configure your application with a file watcher to detect the changed file data and reconfigure itself as needed. The following code shows a Deployment that mounts a ConfigMap and a Secret file as a volume:
apiVersion
:
v1
kind
:
ConfigMap
metadata
:
name
:
nginx-http-config
namespace
:
myapp-prod
data
:
config
:
|
http {
server {
location / {
root /data/html;
}
location /images/ {
root /data;
}
}
}
apiVersion
:
v1
kind
:
Secret
metadata
:
name
:
myapp-api-key
type
:
Opaque
data
:
myapikey
:
YWRtd5thSaW4=
apiVersion
:
apps/v1
kind
:
Deployment
metadata
:
name
:
mywebapp
namespace
:
myapp-prod
spec
:
containers
:
-
name
:
nginx
image
:
nginx
ports
:
-
containerPort
:
8080
volumeMounts
:
-
mountPath
:
/etc/nginx
name
:
nginx-config
-
mountPath
:
/usr/var/nginx/html/keys
name
:
api-key
volumes
:
-
name
:
nginx-config
configMap
:
name
:
nginx-http-config
items
:
-
key
:
config
path
:
nginx.conf
-
name
:
api-key
secret
:
name
:
myapp-api-key
secretname
:
myapikey
Note
There are a couple of things to consider when using volumeMounts
. First, as soon as the ConfigMap/Secret is created, add it as a volume in your pod’s specification. Then mount that volume into the container’s filesystem. Each property name in the ConfigMap/Secret will become a new file in the mounted directory, and the contents of each file will be the value specified in the ConfigMap/Secret. Second, avoid mounting ConfigMaps/Secrets using the volumeMounts.subPath
property. This will prevent the data from being dynamically updated in the volume if you update a ConfigMap/Secret with new data.
-
ConfigMap/Secrets must exist in the namespace for the pods that will consume them prior to the pod being deployed. The optional flag can be used to prevent the pods from not starting if the ConfigMap/Secret is not present.
-
Use an admission controller to ensure specific configuration data or to prevent deployments that do not have specific configuration values set. An example would be if you require all production Java workloads to have certain JVM properties set in production environments. There is an alpha API called
PodPresets
that will allow ConfigMaps and secrets to be applied to all pods based on an annotation, without needing to write a custom admission controller. -
If you’re using Helm to release applications into your environment, you can use a life cycle hook to ensure the ConfigMap/Secret template is deployed before the Deployment is applied.
-
Some applications require their configuration to be applied as a single file such as a JSON or YAML file. ConfigMap/Secrets allows an entire block of raw data by using the
|
symbol, as demonstrated here:
apiVersion
:
v1
kind
:
ConfigMap
metadata
:
name
:
config-file
data
:
config
:
|
{
"iotDevice": {
"name": "remoteValve",
"username": "CC:22:3D:E3:CE:30",
"port": 51826,
"pin": "031-45-154"
}
}
-
If the application uses system environment variables to determine its configuration, you can use the injection of the ConfigMap data to create an environment variable mapping into the pod. There are two main ways to do this: mounting every key/value pair in the ConfigMap as a series of environment variables into the pod using
envFrom
and then usingconfigMapRef
orsecretRef
, or assigning individual keys with their respective values using theconfigMapKeyRef
orsecretKeyRef
. -
If you’re using the
configMapKeyRef
orsecretKeyRef
method, be aware that if the actual key does not exist, this will prevent the pod from starting. -
If you’re loading all of the key/value pairs from the ConfigMap/Secret into the pod using
envFrom
, any keys that are considered invalid environment values will be skipped; however, the pod will be allowed to start. The event for the pod will have an event with reasonInvalidVariableNames
and the appropriate message about which key was skipped. The following code is an example of a Deployment with a ConfigMap and Secret reference as an environment variable:
apiVersion
:
v1
kind
:
ConfigMap
metadata
:
name
:
mysql-config
data
:
mysqldb
:
myappdb1
user
:
mysqluser1
apiVersion
:
v1
kind
:
Secret
metadata
:
name
:
mysql-secret
type
:
Opaque
data
:
rootpassword
:
YWRtJasdhaW4=
userpassword
:
MWYyZDigKJGUyfgKJBmU2N2Rm
apiVersion
:
apps/v1
kind
:
Deployment
metadata
:
name
:
myapp-db-deploy
spec
:
selector
:
matchLabels
:
app
:
myapp-db
template
:
metadata
:
labels
:
app
:
myapp-db
spec
:
containers
:
-
name
:
myapp-db-instance
image
:
mysql
resources
:
limits
:
memory
:
"128Mi"
cpu
:
"500m"
ports
:
-
containerPort
:
3306
env
:
-
name
:
MYSQL_ROOT_PASSWORD
valueFrom
:
secretKeyRef
:
name
:
mysql-secret
key
:
rootpassword
-
name
:
MYSQL_PASSWORD
valueFrom
:
secretKeyRef
:
name
:
mysql-secret
key
:
userpassword
-
name
:
MYSQL_USER
valueFrom
:
configMapKeyRef
:
name
:
mysql-config
key
:
user
-
name
:
MYSQL_DB
valueFrom
:
configMapKeyRef
:
name
:
mysql-config
key
:
mysqldb
-
If there is a need to pass command-line arguments to your containers, environment variable data can be sourced using
$(ENV_KEY)
interpolation syntax:
[
...
]
spec
:
containers
:
-
name
:
load-gen
image
:
busybox
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
"while
true;
do
curl
$(WEB_UI_URL);
sleep
10;done"
]
ports
:
-
containerPort
:
8080
env
:
-
name
:
WEB_UI_URL
valueFrom
:
configMapKeyRef
:
name
:
load-gen-config
key
:
url
-
When consuming ConfigMap/Secret data as environment variables, it is very important to understand that updates to the data in the ConfigMap/Secret will not update in the pod and will require a pod restart either through deleting the pods and letting the ReplicaSet controller create a new pod, or triggering a Deployment update, which will follow the proper application update strategy as declared in the Deployment specification.
-
It is easier to assume that all changes to a ConfigMap/Secret require an update to the entire deployment; this ensures that even if you’re using environment variables or volumes, the code will take the new configuration data. To make this easier, you can use a CI/CD pipeline to update the
name
property of the ConfigMap/Secret and also update the reference in the deployment, which will then trigger an update through normal Kubernetes update strategies of your deployment. We will explore this in the following example code. If you’re using Helm to release your application code into Kubernetes, you can take advantage of an annotation in the Deployment template to check thesha256
checksum of the ConfigMap/Secret. This triggers Helm to update the Deployment using thehelm upgrade
command when the data within a ConfigMap/Secret is changed:
apiVersion
:
apps/v1
kind
:
Deployment
[
...
]
spec
:
template
:
metadata
:
annotations
:
checksum/config
:
{{
include (print $.Template.BasePath "/configmap.yaml") . | sha256sum
}}
[
...
]
Best practices specific to secrets
Because of the nature of sensitive data of the Secrets API, there are naturally more specific best practices, which are mainly around the security of the data itself:
-
The original specification for the Secrets API outlined a pluggable architecture to allow the actual storage of the secret to be configurable based on requirements. Solutions such as HashiCorp Vault, Aqua Security, Twistlock, AWS Secrets Manager, Google Cloud KMS, or Azure Key Vault allow the use of external storage systems for secret data using a higher level of encryption and auditability than what is offered natively in Kubernetes.
-
Assign an
imagePullSecrets
to aserviceaccount
that the pod will use to automatically mount the secret without having to declare it in thepod.spec
. You can patch the default service account for the namespace of your application and add theimagePullSecrets
to it directly. This automatically adds it to all pods in the namespace:
Create the docker-registry secret first kubectl create secret docker-registry registryKey --docker-server myreg.azurecr.io --docker-username myreg --docker-password$up3r$3cr3tP
@ssw0rd --docker-email ignore@dummy.com patch the default serviceaccountfor
the namespace you wish to configure kubectl patch serviceaccount default -p'{"imagePullSecrets": [{"name":
"registryKey"}]}'
-
Use CI/CD capabilities to get secrets from a secure vault or encrypted store with a Hardware Security Module (HSM) during the release pipeline. This allows for separation of duties. Security management teams can create and encrypt the secrets, and developers just need to reference the names of the secret expected. This is also the preferred DevOps process to ensure a more dynamic application delivery process.
RBAC
When working in large, distributed environments, it is very common that some type of security mechanism is needed to prevent unauthorized access to critical systems. There are numerous strategies around how to limit access to resources in computer systems, but the majority all go through the same phases. Using an analogy of a common experience such as flying to a foreign country can help explain the processes that happen in systems like Kubernetes. We can use the common travler’s experience with a passport, travel visa, and customs or border guards to show the process:
-
Passport (subject authentication). Usually you need to have a passport issued by some government agency that will offer some sort of verification as to who you are. This would be equivalent to a user account in Kubernetes. Kubernetes relies on an external authority to authenticate users; however, service accounts are a type of account that is managed directly by Kubernetes.
-
Visa or travel policy (authorization). Countries will have formal agreements to accept travelers holding passports from other countries through formal short-term agreements such as visas. The visas will also outline what the visitor may do and for how long they may stay in the visiting country, depending on the specific type of visa. This would be equivalent to authorization in Kubernetes. Kubernetes has different authorization methods, but the most used is RBAC. This allows very granular access to different API capabilities.
-
Border patrol or customs (admission control). When entering a foreign country, usually there is a body of authority that will check the requisite documents, including the passport and visa, and, in many cases, inspect what is being brought into the country to ensure it abides by that country’s laws. In Kubernetes this is equivalent to admission controllers. Admission controllers can allow, deny, or change the requests into the API based upon rules and policies that are defined. Kubernetes has many built-in admission controllers such as PodSecurity, ResourceQuota, and ServiceAccount controllers. Kubernetes also allows for dynamic controllers through the use of validating or mutating admission controllers.
The focus of this section is the least understood and the most avoided of these three areas: RBAC. Before we outline some of the best practices, we first must present a primer on Kubernetes RBAC.
RBAC Primer
The RBAC process in Kubernetes has three main components that need to be defined: the subject, the rule, and the role binding.
Subjects
The first component is the subject, the item that is actually being checked for access. The subject is usually a user, a service account, or a group. As mentioned earlier, users as well as groups are handled outside of Kubernetes by the authorization module used. We can categorize these as basic authentication, x.509 client certificates, or bearer tokens. The most common implementations use either x.509 client certificates or some type of bearer token using something like an OpenID Connect system such as Azure Active Directory (Azure AD), Salesforce, or Google.
Note
Service accounts in Kubernetes are different than user accounts in that they are namespace bound, internally stored in Kubernetes; they are meant to represent processes, not people, and are managed by native Kubernetes controllers.
Rules
Simply stated, this is the actual list of actions that can be performed
on a specific object (resource) or a group of objects in the API. Verbs
align to typical CRUD (Create, Read, Update, and Delete) type operations
but with some added capabilities in Kubernetes such as watch
, list
, and
exec
. The objects align to the different API components and are grouped
together in categories. Pod objects, as an example, are part of the core
API and can be referenced with apiGroup: ""
whereas deployments are under
the app API Group. This is the real power of the RBAC process and
probably what intimidates and confuses people when creating proper RBAC controls.
Roles
Roles allow the definition of scope of the rules defined. Kubernetes has
two types of roles, role
and clusterRole
, the difference being that
role
is specific to a namespace, and clusterRole
is a cluster-wide
role across all namespaces. An example Role definition with namespace scope would be as follows:
kind
:
Role
apiVersion
:
rbac.authorization.k8s.io/v1
metadata
:
namespace
:
default
name
:
pod-viewer
rules
:
-
apiGroups
:
[
""
]
# "" indicates the core API group
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]
RoleBindings
The RoleBinding allows a mapping of a subject like a user or group to a
specific role. Bindings also have two modes: roleBinding
, which is
specific to a namespace, and clusterRoleBinding
, which is across the
entire cluster. Here’s an example RoleBinding with namespace scope:
kind
:
RoleBinding
apiVersion
:
rbac.authorization.k8s.io/v1
metadata
:
name
:
noc-helpdesk-view
namespace
:
default
subjects
:
-
kind
:
User
name
:
helpdeskuser@example.com
apiGroup
:
rbac.authorization.k8s.io
roleRef
:
kind
:
Role
#this must be Role or ClusterRole
name
:
pod-viewer
# this must match the name of the Role or ClusterRole to bind to
apiGroup
:
rbac.authorization.k8s.io
RBAC Best Practices
RBAC is a critical component of running a secure, dependable, and stable Kubernetes environment. The concepts underlying RBAC can be complex; however, adhering to a few best practices can ease some of the major stumbling blocks:
-
Applications that are developed to run in Kubernetes rarely ever need an RBAC role and role binding associated to it. Only if the application code actually interacts directly with the Kubernetes API directly does the application require RBAC configuration.
-
If the application does need to directly access the Kubernetes API to perhaps change configuration depending on endpoints being added to a service, or if it needs to list all of the pods in a specific namespace, the best practice is to create a new service account that is then specified in the pod specification. Then, create a role that has the least amount of privileges needed to accomplish its goal.
-
Use an OpenID Connect service that enables identity management and, if needed, two-factor authentication. This will allow for a higher level of identity authentication. Map user groups to roles that have the least amount of privileges needed to accomplish the job.
-
Along with the aforementioned practice, you should use Just in Time (JIT) access systems to allow site reliability engineers (SREs), operators, and those who might need to have escalated privileges for a short period of time to accomplish a very specific task. Alternatively, these users should have different identities that are more heavily audited for sign-on, and those accounts should have more elevated privileges assigned by the user account or group bound to a role.
-
Specific service accounts should be used for CI/CD tools that deploy into your Kubernetes clusters. This ensures for auditability within the cluster and an understanding of who might have deployed or deleted any objects in a cluster.
-
If you’re using Helm to deploy applications, the default service account is Tiller, deployed to
kube-system
. It is better to deploy Tiller into each namespace with a service account specifically for Tiller that is scoped for that namespace. In the CI/CD tool that calls the Helm install/upgrade command, as a prestep, initialize the Helm client with the service account and the specific namespace for the deployment. The service account name can be the same for each namespace, but the namespace should be specific. It is important to call out that as of this publication, Helm v3 is in alpha state and one of its core principles is that Tiller is no longer needed to run in a cluster. An example Helm Init with a Service account and namespace would look like this:
kubectl create namespace myapp-prod kubectl create serviceaccount tiller --namespace myapp-prod cat<<EOF | kubectl apply -f -
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tiller
namespace: myapp-prod
rules:
- apiGroups: ["", "batch", "extensions", "apps"]
resources: ["*"]
verbs: ["*"]
EOF
cat<<EOF | kubectl apply -f -
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tiller-binding
namespace: myapp-prod
subjects:
- kind: ServiceAccount
name: tiller
namespace: myapp-prod
roleRef:
kind: Role
name: tiller
apiGroup: rbac.authorization.k8s.io
EOF
helm init --service-account=
tiller --tiller-namespace=
myapp-prod helm install ./myChart --name myApp --namespace myapp-prod --set global.namespace=
myapp-prod
Note
Some public Helm charts do not have value entries for namespace choices to deploy the application components. This might require customization of the Helm chart directly or using an elevated Tiller account that can deploy to any namespace and has rights to create namespaces.
-
Limit any applications that require
watch
andlist
on the Secrets API. This basically allows the application or the person who deployed the pod to view the secrets in that namespace. If an application needs to access the Secrets API for specific secrets, limit usingget
on any specific secrets that the application needs to read outside of those that it is directly assigned.
Summary
Principles for developing applications for cloud native delivery is a topic for another day, but it is universally accepted that strict separation of configuration from code is a key principal for success. With native objects for nonsensitive data, the ConfigMap API, and for sensitive data, the Secrets API, Kubernetes can now manage this process in a declarative approach. As more and more critical data is represented and stored natively in the Kubernetes API, it is critical to secure access to those APIs through proper gated security processes such as RBAC and integrated authentication systems.
As you’ll see throughout the rest of this book, these principles permeate every aspect of the proper deployment of services into a Kubernetes platform to build a stable, reliable, secure, and robust system.
Get Kubernetes Best Practices now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.