Argo CD: Up and Running

Chapter 4. Synchronizing Applications

A Note for Early Release Readers

With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles.

This will be the 5th chapter of the final book

If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at [e-mail].

Argo CD makes it easy to be able to take Kubernetes resources stored within Git or Helm repositories and apply them to a target cluster – a process known as synchronization. Given that this capability is one of the core features of Argo CD, there are a variety of options available for determining when the synchronization process will be triggered and how the Kubernetes resources will be applied. This level of control is important as there may be a need to guard exactly how and when content is applied. In this chapter, we will explore the options available when synchronizing Argo CD Applications, their impact against the lifecycle of the Application itself, the Argo CD Server, and ultimately the target Kubernetes cluster.

Managing how Applications are Synchronized

Given that the synchronization of content from source to target Kubernetes cluster is a fundamental concept in Argo CD, it is important to first understand the defaults that Argo CD applies and the various levels of customizations that are available. If you recall back to Chapter 4 - Managing Applications, synchronization was one of the topics that were briefly introduced including how the configurations can be defined within the .spec.syncPolicy property of an Application.

By default, when Applications are created, none of the rendered resources are applied to the Kubernetes cluster. This may surprise many new Argo CD users given that Argo CD is a tool that manages assets that are destined for Kubernetes. However, there are a number of reasons why this is Argo CD’s default behavior:

• As the configurations for an Application are refined, there may be a need or desire to “preview” the changes that would be applied without performing any change

• A desire for control of when and how resource are applied

• Organizational policies prohibit automating changes to infrastructure

The Argo CD user interface and Application resource do provide a glimpse of the resources that would be affected, but any synchronization against the cluster would need to be performed in a manual fashion. Synchronization of manifests can be achieved using the user interface by selecting the “Sync” button on the Application or from the Argo CD CLI using the argocd app sync command.

But, since most users would want to take advantage of an automated synchronization of an Application, let’s illustrate the ways that this can be achieved:

1. Specifying the sync policy for the Application using the argocd CLI.

   $ argocd app set <APPNAME> --sync-policy automated

2. Selecting the “Enable Auto-Sync” button within the Argo CD user interface

3. Defining the configuration explicitly within the Application resource

   spec:
     syncPolicy:
       automated: {}

Regardless of the option chosen, as soon as the source content differs from the live state of the cluster, the Application will be synchronized.

Sync options

Aside from the fundamental determination of whether an Application should be synchronized automatically or manually, Argo CD can be configured to perform a customized operation of how it synchronizes the desired state to the target cluster through the .spec.syncPolicy.syncOptions property. These customizations can, for the most part, be configured on the Application resource itself. However, others can be defined as annotations within each individual resource that is associated with an Application. This is especially useful when you want a specific action to occur against a set of resources, but not all of the manifests associated within an Argo CD Application.

Let’s first take a look at the how Sync Options can be used within an Argo CD application

Application Level Options

As mentioned previously, synchronization options are specified under the .spec.syncPolicy.syncOptions in the Application Manifest. These options will affect all resources that are associated with f the Argo CD Application.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
    name: sample-app
    namespace: argocd
spec:
  syncPolicy:
    syncOptions:
      - Validate=true
      - ApplyOutOfSyncOnly=true
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
      - Replace=false
      - ServerSideApply=true
      - FailOnSharedResource=true
      - RespectIgnoreDifferences=true

Let’s dive a little deeper into the syncOptions configurations:

Validate=false

By default, Argo CD uses Kubernetes API validation and will fail the sync operation if the manifest is not valid (equivalent to running: kubectl apply --validate=false ). The default value is: true

ApplyOutOfSyncOnly=true

By default, Argo CD applies every object in an Argo CD Application. This could pose a problem if you have thousands and thousands of objects. This option only synchronizes/applies objects that are out of sync.

CreateNamespace=true

This option creates the namespace (under in the spec.destination.namespace section of the Argo CD Application), if it does not already exist, before Argo CD attempts to apply the objects in an Application.

PrunePropagationPolicy=foreground –

This option shapes how the Application handles pruning/deleting of resources (known as garbage collection). The default is foreground and other options available are background and orphan. To learn more about how Kubernetes handles garbage collection, Read the upstream Kubernetes documentation.

PruneLast=true

This option allows the ability for resource pruning to happen as a final part of a sync operation, after the other resources have been deployed and become healthy, and after all other waves are completed successfully.

Replace=false

Argo CD, by default, does the equivalent of kubectl apply. This sometimes poses an issue when the object is too big to fit into kubectl.kubernetes.io/last-applied-configuration annotation. Note, this option could be dangerous, and be used with caution.

ServerSideApply=true

This option enables Argo CD to use server side apply when running a sync operation. This is equivalent to running: kubectl apply --server-side. Most of the time, since this option is used to apply deltas of changes, the Validate=false option is frequently used in conjunction with this option.

FailOnSharedResource=true

With this option, Argo CD will mark the Application as “failed” whenever it finds a resource associated with the Application that has already been applied in the cluster via another Application.

RespectIgnoreDifferences=true

By default, Argo CD uses the ignoreDifferences config, found in .spec.ignoreDifferences, only for calculating the difference between the live and desired state (but still applies the object as it is defined” in Git). This option also takes it into consideration during the sync operation.

Resource Level Options

Along with the sync options on the Argo CD Application level, users can also apply these configurations/options at the object/individual resource level. Meaning that you don’t have to apply any of the sync options against all resources contained within the entire Argo CD Application, but to only specific objects. A subset of the Application sync options are available to individual objects, as well as several other additional options.

These resource level options can be set by annotating the resource you want the option to apply to. You can do this by defining the metadata.annotations.argocd.argoproj.io/sync-options annotation on the resource you would like to apply the option to. For example, to skip Kubernetes validation on a specific object:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Validate=false

By implementing this approach, only the object with this annotation will skip Kubernetes validation while the rest of the objects within the Argo CD Application will be validated. The options available via the argocd.argoproj.io/sync-options annotation are:

• Validate

• PruneLast

• Replace

• ServerSideApply

In addition, the following options are available for individual resources using the argocd.argoproj.io/sync-options annotation:

Prune=false

This prevents the annotated object from being pruned.

SkipDryRunOnMissingResource=true

Argo CD, by default, performs a “dry run” of applying the manifests (equivalent to using the --dry-run option with kubectl); this option skips the dry run step. This is especially useful if you are deploying CRDs or Operators as the associated resource may not be available as a registered resource at the specific validation time. This option is commonly paired with the retry strategy which will perform subsequent attempts to synchronize the Application where a failure no longer occurs as the desired resource has become available.

Users can specify multiple options in the annotation by separating the options with a comma (,) between each of the desired options. For example; to disable validation and use server side apply within a resource, you can set the following in your object:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Validate=false,ServerSideApply=true

Using the above configuration, the object with this annotation will disable validation and use server side apply.

Sync Order and Hooks

Argo CD has the ability to customize the order in which the manifests are applied. Furthermore, Argo CD incorporates different sync phases so that users can further fine tune how objects are applied to the target cluster.

Hooks

Argo CD has the ability to set up different sync phases by allowing the user to utilize “hooks”. These injection points within the Application lifecycle enable additional automation, such as running scripts before, during, and/or after a sync has completed, to supplement applying the standard set of resources. You can also use hooks in the event a sync has failed for whatever reason. While hooks can be implemented as any Kubernetes object, they are usually Pods or Jobs.

There are 4 hooks that can be used in your Argo CD Sync process:

PreSync

This phase occurs prior to the Sync phase. This is typically used for actions that need to occur before the Application is synced. A common use case is running a script that performs a schema update against a database.

Sync

This is the “standard” or “default” phase for Argo CD and is executed once the PreSync phase has finished. This is typically used to aid the Argo CD Application deployment process in the event more complex activities within the Application needs to occur.

PostSync

This phase occurs after the Sync phase has been completed. This can be used to send a notification that the phase has been completed or to trigger a CI progress or continue a CI/CD workflow.

SyncFail

This is a special hook that is run only if a sync operation has failed. This is normally used for alerting or performing cleanup activities.

When setting up an Argo CD Application, the resources that are in your source of truth are applied to the destination cluster during the Sync phase. The other phases are used to perform pre or post tasks before and/or after the objects are applied in the Sync phase.

It is important to note that each phase is dependent on the success of the previous phase (with the exception of the SyncFail phase). For example: if an error occurs in the PreSync phase, the Sync phase will not run.

In order to indicate which resource in your Git repository belongs to which phase, you will have to annotate the desired resource with argocd.argoproj.io/hook with the value of the phase that it should execute within (the absence of the hook annotation results in the resource being applied during the Sync phase). For example, a Job to be executed in the PostSync phase, the following annotation is applied:

metadata:
  annotations:
    argocd.argoproj.io/hook: PostSync

Hook Deletion Policies

Resources that make use of Hooks can be deleted when a sync operation is performed by using the argocd.argoproj.io/hook-delete-policy annotation. The following hook deletion policies are available.

HookSucceeded

The hook resource/object is deleted once it has successfully completed.

HookFailed

The hook resource/object is deleted if the hook has failed.

BeforeHookCreation

Any hook resource/object will be deleted before the new one is created.

Here is an example of a PostSync hook with a deletion policy of HookSucceeded.

metadata:
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

It is important to note that hooks that are named (i.e. ones with .metadata.name defined) will be created/run only once. If you want a hook to be re-created or re-ran each time there is a sync operation; either use the BeforeHookCreation deletion policy or use .metadata.generateName in your resource/object.

Note

As of the time of this writing, certain tools, such as Kustomize, have limited support for the use of the generateName property.

Sync waves

Argo CD applies manifests in a specific order. You can see this order by inspecting the code.In most cases, the default order that Argo CD applies resources should work in most situations. However, complex deployments may inevitably require changes to this default order. This is where Syncwaves come in.

The concept of Syncwaves is pretty straightforward. The desired resource is annotated with the order in which you wish Ago CD to apply your manifests using argocd.argoproj.io/sync-wave key with an integer value denoted as a string.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "5"

By default, every resource gets assigned “wave 0”, unless otherwise specified via the annotation. Numbers can be negative as well. So, for example, consider the following:

• Namespace as wave “-1”

• Service Account as wave “0”

• Deployment as wave “1”

The Namespace would be applied first, then the Service Account, and then finally the Deployment.

A good use case for Syncwaves is to applyCustom Resource Definitions first before the corresponding Custom Resource.

Using Syncwaves within Hooks

Syncwaves can also be used within the confines of a Hook. Meaning you can have resources within a PreSync Hook Phase be applied in a specific order, within that order, without affecting other Hook Phases. In the following example, the Job will be applied in wave “3” within the PreSync Hook Phase.

apiVersion: batch/v1
kind: Job
metadata:
  name: create-tables
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/hook: PreSync

Now, you can also have the following resource in a PostSync hook

apiVersion: batch/v1
kind: Job
metadata:
  name: test-deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
    argocd.argoproj.io/hook: PostSync

In these two examples, the create-tables will be applied before the test-deployment even though test-deployment is a lower “wave. This is due to the fact that the create-tables resource is in a different Hook Phase. The important thing to note when considering Syncwaves with Hooks is that Syncwaves are scooped within each Hook Phase.

Compare options

There might be cases where you will need to exclude resources from the overall status of your application. For example, if you have a resource created by another controller (this is common when working with Operators. You can read more about Operators here). This can be achieved with the following annotation.

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous

Note

This only affects the sync status. If the resource’s health is degraded, then the Application will also be degraded.

For example, the following Secret instructs the OpenShift OAuth Operator to create another Secret for the OpenShift OAuth Controller to consume. By doing so, Argo CD will mark your Argo CD Application “Out of Sync”. To work around this issue, use the aforementioned argocd.argoproj.io/compare-options: IgnoreExtraneous annotation.

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: htpass-secret
  namespace: openshift-config
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous
data:
  htpasswd: bm90VGhlRHJvaWRzWW91cmVMb29raW5nRm9y

This will mark your Application as “Healthy” in Argo CD, but it’s important to note that it’ll mark the created resource as “out of sync”. However, the health is not affected.

You can also ignore certain aspects during diffing and syncing, this will be covered in the following section titled “Managing Resource Differences.”

Managing Resource Differences

Argo CD allows you to manage how you handle differences from your source of truth and current state within Kubernetes by the way of ignoring differences. There are several locations where ignoring differences can be configured. This configuration can be applied on a per Argo CD Application basis or for the whole Argo CD system (where all the Applications in an Argo CD installation are affected)

Application Level Diffing

As the name suggests, Application Level Diffing allows you to ignore differences within individual Applications at a specific JSON path, using RFC6902 JSON patches and JQ path expressions. Using the JSON path, you can specify paths referencing properties Argo CD should ignore when it compares the running state with the desired state defined. Here is an example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jsonPointers:
    - /spec/replicas

The ignoreDifferences setting allows you to specify the name of the resource and the namespace as well as the GVK (Group Version Kind). For more complex manifests, you can use the JQ path expression to define specific items to ignore in a more granular fashion. For example:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
spec:
  ignoreDifferences:
  - group: apps
    kind: Deployment
    jqPathExpressions:
    - .spec.template.spec.initContainers[] | select(.name == "injected-init-container")

You can also ignore fields owned by specific managers by using managedFieldsManagers and listing the specific managers to ignore.

An additional item to note.; Most users will use the RespectIgnoreDifferences sync option in conjunction with this ignoreDifferences setting..

System Level Diffing

Argo CD can also be set up to ignore differences at a system level. This allows administrators to be able to set global ignore settings for the specific Argo CD installation. These configurations can be set up for a specified group and kind by using the resource.customizations key of argocd-cm ConfigMap using the following format.

data:
  resource.customizations.ignoreDifferences.apps_Deployment: |
   jsonPointers:
    - /spec/replicas

Take note the resource.customizations key also includes the keyword ignoreDifferences with the GKV demarcated by an underscore (_) using a flattened approach. For more information about how to formulate these settings please see the official Argo CD documentation site on system level diffing.

Use Case: Database Schema Setup

With an understanding of some of the ways to customize the synchronization and the associating current state for applications, let’s see it in action with one of the most common use cases: A Database Schema Setup..

We are going to be deploying an Application that is going to consist of a backend database. The Database will be set up at deploy time, which means that the database schema will need to be loaded as a part of the deployment. Furthermore, the database schema setup needs to run after the database is up and running. For this specific use case we are going to be making use of SyncWaves and Argo CD Application SyncOptions.

Argo CD Application Overview

All the artifacts we will be using are in the aforementioned companion repo that you can find at https://github.com/sabre1041/argocd-up-and-running-book - make sure you’ve cloned this repository if you have not done so already and ensure that you are in the root directory of this repository.

Inspect the Argo CD Application for this use case which is located in the ch05 directory.

Execute cat ch05/pricelist-app.yaml from the root directory of the repository and you will see the following manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pricelist-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.iospec:
  project: default
  source:
    path: ch05/manifests/
    repoURL: https://github.com/sabre1041/argocd-up-and-running-book
    targetRevision: main
  destination:
    namespace: pricelist
    name: in-cluster
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

This manifest should look familiar if you have already completed Chapter 4 - Managing Applications. There are several items of note to point out:

• The .spec.syncPolicy has the automated options of prue: true and selfHeal: true. This means that Argo CD will synchronize this Application automatically whenever it’s out of sync. In addition, it will also delete resources that it is not keeping track of.

• Under .spec.syncPolicy, the CreateNamespace=true option under syncOptions is also defined which specifies that Argo CD will create the destination Namespace if it doesn’t already exist.

• Retries under that .spec.syncPolicy.retry property have also been defined.

One final item to note is that Argo CD will be deploying manifests under the ch05/manifests/ directory from the repository as denoted in the .spec.source.path section. This last item is what we will cover in the next section.

Manifest Syncwave Overview

If you take a look under the ch05/manifests/ directory, you will see a kustomization.yaml file, which for the purposes of this example, aggregates the manifests that need to be applied. It’s a simple list; basically, it is the resources that we want applied to the cluster.

Note

For more information about Kustomize, please see Chapter 4 - Managing Applications

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: pricelist
resources:
- pricelist-db-pvc.yaml
- pricelist-db-svc.yaml
- pricelist-db.yaml
- pricelist-deploy.yaml
- pricelist-job.yaml
- pricelist-svc.yaml

Normally, Argo CD would apply these manifests in the same order as the output of kustomize build in this directory. However, we’ve added a syncwave annotation to customize the order Argo CD should apply these manifests.

Prior to any other resource in this Application being applied, we want the database and any backend storage to be up and running first. Therefore, we’ve annotated the pricelist-db-pvc.yaml (Persistent Volume Claim for the Database) and pricelist-db.yaml (Database Deployment) manifests with the argocd.argoproj.io/sync-wave: “1” annotation to denote that we want these two manifests to be applied first. They both should have the following annotation.

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"

This will not only make Argo CD apply these manifests first, but the annotation also causes Argo CD to wait until these manifests are in a “Ready” state before attempting to go on the next manifest. Once all the manifests in wave 1 are applied and reporting a Ready state, the next wave is applied.

In our use case, the next wave is the pricelist-db-svc.yaml file, which has the argocd.argoproj.io/sync-wave: “2” annotation:

apiVersion: v1
kind: Service
metadata:
  name: mysql
  annotations:
    argocd.argoproj.io/sync-wave: "2"

Since this is the only manifest with that sync wave annotation, this pricelist-db-svc.yaml file will be applied after wave 1.

You can inspect the other manifests in the ch05/manifests/ directory to inspect the order that they will be applied in.

• pricelist-db-pvc.yaml and pricelist-db.yaml as syncwave 1

• pricelist-db-svc.yaml as syncwave 2

• pricelist-deploy.yaml as syncwave 3

• pricelist-svc.yaml as syncwave 4

• pricelist-job.yaml in a PostSync hook in syncwave 0

Before moving on, it’s important to note that when you inspect the pricelist-job.yaml manifest, this Job is responsible for setting up the Database schema. This Job also runs as a PostSync hook, which means that it will be applied after all the manifests in the Sync phase have been applied. Also note that the Job has a syncwave of 0. Although a syncwave of 0 is the default, the annotation was added to illustrate that syncwaves work within phases. Taking a look at the annotations in the pricelist-job.yaml manifest:

apiVersion: batch/v1
kind: Job
metadata:
  name: pricelist-postdeploy
  annotations:
    argocd.argoproj.io/sync-wave: "0"
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation

Another important item to note is the use of a hook deletion policy. This annotation ensures that this Job object should be deleted before the hook phase starts in subsequent sync runs if it is present. To learn more about hook deletion policies, please consult the official Argo CD documentation on resource hooks.

Importance of Probes

Argo CD uses several different sources to determine the overall health of the Application being deployed. One of the important metrics used is health status from the Kubernetes API. In order for this capability to be utilized, it’s very important to have readiness/liveness probes set up correctly for each object that needs it.

Note

For more information about how Argo CD handles Application health please consult the official documentation on Application health.

In our particular use case, the resources that require probes to be defined in order to achieve the desired goal are the Database Deployment and the web app Deployment. Taking a look at the pricelist-db.yaml file, you’ll see the following probes.

spec:
  template:
    spec:
      containers:
      - image: registry.access.redhat.com/rhscl/mysql-56-rhel7:latest
        name: mysql
        livenessProbe:
          tcpSocket:
            port: 3306
          initialDelaySeconds: 12
          periodSeconds: 10
        readinessProbe:
          tcpSocket:
            port: 3306
          initialDelaySeconds: 12
          periodSeconds: 10

In this instance, TCP port 3306 is waiting to become active before considering the database deployment alive and ready to receive requests. For the web app, which is the pricelist-deploy.yaml file, you will see the following probes configured.

spec:
  template:
    spec:
      containers:
      - image: quay.io/redhatworkshops/pricelist:latest
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 2
        livenessProbe:
          tcpSocket:
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 2

In the web app Deployment, we are considering the web app alive when TCP port 8080 is active. The app will not be considered ready until an HTTP GET request returns a response code of 200 on port 8080.

In both cases (the database Deployment and web app Deployment), both probes need to be successful before Argo CD considers the Application to be “healthy” and “synced”.

Note

For more information on probes and how to set them up, please see the official Kubernetes documentation on probes.

Seeing It In Action

Now that we’ve reviewed the use case in detail, let’s see it in action by using these manifests in our KIND instance. From the root directory of the companion git repository, apply the Application manifest by running the following command:

kubectl apply -f ch05/pricelist-app.yaml

An Argo CD Application tile should appear in the Argo CD UI as a result. The tile will appear similar to what is depicted in Figure 4-1.

Figure 4-1. Pricelist Application Tile

The first thing Argo CD does is apply the first syncwave, which is our storage and database Deployment. After clicking on the Application tile, you should be able to see these resources enter the syncing phase first while the other resources are in the “missing” state. Take a look at Figure 4-2 for an example on how this is displayed.

Figure 4-2. Pricelist Syncwave 1

When the storage is provisioned and the MySQL database is deployed, the next object that Argo CD will apply in our use case is the MySQL service. The Application overview will appear similar to Figure 4-3:

Figure 4-3. Pricelist Syncwave 2

After the service is healthy, Argo CD will apply the web app Deployment as seen in Figure 4-4:

Figure 4-4. Pricelist Syncwave 3

Once the web app is deployed, the Service for the web app is applied as denoted in Figure 4-5:

Figure 4-5. Pricelist Syncwave 4

Once the web app Service is deployed and in a healthy state; the Sync phase is considered complete and Argo CD will enter the PostSync phase The final step that Argo CD performs is applying the Job that facilitates the Database schema setup. In the Argo CD UI, this is indicated by an anchor (⚓) symbol within the Job, as seen in Figure 4-6:

Figure 4-6. Pricelist PostSync Hook

Once the PostSync phase finishes, you should now see the Argo CD Application tile for the Application show “Healthy” and “Synced” status in the Application overview page. See Figure 4-7 for how this appears:

Figure 4-7. Pricelist Synced and Healthy

Summary

In this chapter, we covered how Argo CD synchronizes Applications and how you can customize the method in which Argo CD performs synchronizations on the individual Application level, and the system as a whole. We also reviewed how to further refine your synchronizations by implementing ordering with Syncwaves and Sync Hooks. Finally, we reviewed in detail a use case where Syncwaves and Sync Hooks were used to perform a database schema setup during an Argo CD deployment of an Application. In the next chapter, we will introduce how Argo CD handles Multi-tenancy and how to configure Argo CD Projects to provide those layers of demarcation.