"Oops, something is wrong with your GitOps application!"

GitOps – Operations by Pull Request

GitOps, a natural evolution of the DevOps practice, has gained popularity ever since it was introduced by Weaveworks in 2017. The core idea of GitOps is to treat “everything” as code. What does it mean? You will need to describe the desired state of your system declaratively as code, for example, the infrastructure, configuration, policies, or anything else you could imagine. The declared state of the system is stored and versioned in a Git repository. Naturally, every operational change is made through a typical Git pull request workflow. If you haven’t heard about GitOps, check out more details at What is GitOps.

OpenShift offers a simple solution to help your team adopt a GitOps approach via an operator – OpenShift GitOps. You can find more details in Introduction to GitOps with OpenShift. It uses the upstream project Argo CD as a controller. It continuously monitors the cluster and constantly compares the difference between the declared state from the Git repository and the observed state from the cluster. Whenever there is a drift detected, it can automatically reconcile the difference so that you always have the actual state equal to the desired state. Sounds awesome, right? You saved yourself the trouble of applying changes every single time, but what if the controller failed to synchronize the change? You want to be informed immediately to find out what went wrong. If you are looking for a solution for this, keep reading!

Tell me what you see — Argo CD Notifications!

Essentially, Argo CD Notifications is a controller that monitors Argo CD applications and notifies users about the status of syncs and the health of applications. You can configure it to deliver notifications to receivers like Slack, Email, or custom webhooks. In this article, we are going to set up Argo CD Notifications with Slack using OpenShift GitOps.

Getting Ready!

First of all, we need to create a Slack app for notifications. There is detailed documentation that explains how to configure the Slack app and get the OAuth token. Remember to add appropriate scopes to the Slack app, otherwise, we might have trouble receiving notifications.

After the app is configured, we need to add it to a public or private Slack channel, so it can send notifications to this channel after we set up the notifications controller.

Now we need an OpenShift cluster with OpenShift GitOps installed. If you don’t have a cluster, feel free to use Code Ready Containers or OpenShift Playground from Red Hat to get some hands-on experience. You can easily find the OpenShift GitOps Operator on OperatorHub.

We will perform most of the processes in the terminal, so you might need to install some CLI (e.g. oc, helm) if they are not ready in your terminal.

But wait, how to store the secret?

No one wants to expose sensitive data, but how to handle secrets like a pro in GitOps? There are many ways to achieve secret management. In this article, we are going to store our Slack token in HashiCorp Vault, then we will use argocd-vault-plugin to retrieve and inject the token into our notification application.

First of all, we need to install the vault on our cluster.

$ oc new-project vault
$ oc project vault

$ helm repo add hashicorp https://helm.releases.hashicorp.com

$ helm install vault hashicorp/vault \
  --set "global.openshift=true" \
  --set "server.dev.enabled=true"Code language: JavaScript (javascript)

Then we will need to complete a few steps to create a fully functional vault environment. First, we need to enable Kubernetes Authentication and write the Kubernetes config.

$ oc -n vault rsh vault-0 

$ vault auth enable kubernetes

$ vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \ 
kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Code language: JavaScript (javascript)

Now we can put the Slack token that we received earlier into the vault.

$ vault kv put secret/vplugin/slacktoken slack-token="value"Code language: JavaScript (javascript)

Let’s do a double-check to make sure that the token is correct.

$ vault kv get secret/vplugin/slacktokenCode language: JavaScript (javascript)

The following command is to create the policy for the secret.

$ vault policy write vplugin - <<EOF
path "secret/data/vplugin/slacktoken" {
capabilities = ["read"]
}
EOFCode language: JavaScript (javascript)

The final step is to create an authentication role. We will need to create the ServiceAccount (vplugin) in the namespace (argocd-01) where we will install our Argo CD instance.

$ vault write auth/kubernetes/role/vplugin \
bound_service_account_names=vplugin \
bound_service_account_namespaces=argocd-01 \
policies=vplugin ttl=24h

You can check out this Git repository to get a full picture of how to interact with HashiCorp Vault using argocd-vault-plugin.

Argo CD Configuration

Since we already have the OpenShift GitOps operator installed. It’s time to create an Argo CD instance. First thing first, we need to create a namespace for it.

$ oc new-project argocd-01
$ oc project argocd-01Code language: JavaScript (javascript)

Don’t forget to create the ServiceAccount for the argocd-vault-plugin!

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vplugin
  namespace: argocd-01Code language: PHP (php)

The following is a basic definition of an Argo CD instance – argocd-01, we need to add some ingredients to the installation of argocd-vault-plugin. This goes beyond the scope of this article, but you might want to set up a proper RBAC for your Argo CD as the default policy is read-only. Please check out the Argo CD RBAC documentation.

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: argocd-01
  namespace: argocd-01  
spec:
  applicationInstanceLabelKey: argocd.argoproj.io/argocd-01
  dex:
    openShiftOAuth: true
  rbac:    
    policy: g, argocdadmins, role:admin
    scopes: '[groups]'      
  server:
    route:
      enabled: trueCode language: JavaScript (javascript)

There are a couple of ways to install the argocd-vault-plugin. In this article, we will take a look at how to install the plugin on the Argo CD repo server via an initContainer and a volumeMount. Using the initContainers resource, we can specify the installation command and then allocate a volume to be mounted. At this point, we are provisioning the volume vault-plugin, mounting it to an initContainer ‘install-vault’ on the path /vault-plugin. Then, we mount this volume again to an executable path /usr/local/bin/argocd-vault-plugin.

spec:
  repo:
    initContainers:
      - command:
          - /bin/sh
          - '-c'
          - >-
            curl -v  -LJ --tls-max 1.2
          https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v1.8.0/argocd-vault-plugin_1.8.0_linux_amd64
            -o argocd-vault-plugin && chmod -v +x argocd-vault-plugin
        image: 'registry.redhat.io/ubi8/ubi:latest'
        name: install-vault
        volumeMounts:
          - mountPath: /vault-plugin
            name: vault-plugin
        workingDir: /vault-plugin
    volumes:
      - emptyDir: {}
        name: vault-plugin  
    volumeMounts:
      - mountPath: /usr/local/bin/argocd-vault-plugin
        name: vault-plugin
        subPath: argocd-vault-pluginCode language: PHP (php)

The authentication for HashiCorp Vault is using the ServiceAccount defined in the “repo” Pod of Argo CD. We need to “tell” the operator to mount that ServiceAccountToken to use it for authentication.

repo:
    mountsatoken: true
    serviceaccount: vpluginCode language: JavaScript (javascript)

Wait, wait, we are not done yet. We also need to define the configManagementPlugins. If you want to use Kustomize along with argocd-vault-plugin, remember to register the plugin like this:

configManagementPlugins: |-
    - name: argocd-vault-plugin
      generate:
        command: ["sh", "-c"]
        args: ["kustomize build . | argocd-vault-plugin generate -"]Code language: JavaScript (javascript)

You can find a complete definition of the Argo CD manifest here. If you got some errors during applying the Argo CD manifest, don’t worry, try to delete and re-apply the full Argo CD resource.

argocd-notifications configuration

After we created the argocd-01, it’s time to configure the argocd-notifications controller. First, let’s talk about the secret. We need to create a secret that refers to the Slack token stored in the vault for later use. Remember, <slack-token> is the key of the secret you created in the vault. A small reminder: if you didn’t use base64 to encode the secret, you should keep the stringData:, otherwise, you should use data: to describe your secret.

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  annotations:
    avp.kubernetes.io/path: "secret/data/vplugin/slacktoken"
stringData: 
  slack-token: <slack-token>
type: OpaqueCode language: JavaScript (javascript)

Alright, we’ve managed the first step! Now we need to create a ConfigMap for the controller. We will break it down into a few sections. First, the service used for notifications (in this case, Slack) must be registered in the ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.slack: |
    token: $slack-tokenCode language: PHP (php)

The trigger is an important part of ConfigMap. It defines the conditions under which the notification should be sent. The definition includes the name of the trigger, condition ‘when’, and notification templates. For example, the trigger on sync failed looks like this:

trigger.on-sync-failed: |
    - description: Application syncing has failed
      send:
      - app-sync-failed
      when: app.status.operationState.phase in [‘Error’, ‘Failed’]Code language: JavaScript (javascript)

An important property of the trigger is ‘send’. It contains a list of notification templates. The notification template is also configured in the ConfigMap. It is responsible for generating the notification content. The following is a simple template on ‘app-sync-failed’:

template.app-sync-failed: |
    message: |
      {{if eq .serviceType "slack"}}:exclamation:{{end}} 
      The sync operation of application {{.app.metadata.name}} 
      has failed at {{.app.status.operationState.finishedAt}} 
      with the following error: {{.app.status.operationState.message}}.Code language: JavaScript (javascript)

The subscription to Argo CD application events can be defined using notifications.argoproj.io/subscribe.<trigger>.<service>: <recipient> annotation. For example, the following annotation subscribes to a Slack channel and will send notifications when an Argo CD application is successfully synchronized (or failed to synchronize).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: my-channel
    notifications.argoproj.io/subscribe.on-sync-failed.slack: my-channel

The subscriptions can be configured globally in the ConfigMap using the ‘subscriptions’ field. For example, you can configure the subscription for ‘on-sync-failed’ as shown here:

subscriptions: |
   # subscription for on-sync-failed trigger notifications
    - recipients:
      - slack: <slack-channel>
      triggers:
      - on-sync-failedCode language: HTML, XML (xml)

A complete definition of ConfigMap can be found here.

We still need a few things to complete the configuration, check out the complete argocd-notifications manifest. You can put all the YAML files together in one folder on your Git repository, so later we can deploy it on Argo CD.

Deploy the notifications controller

Finally, the most exciting thing is going to happen. Let’s deploy the notifications controller on argocd-01. We can use the plugin in the application definition:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd-notifications
  namespace: argocd-01
  annotations:
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: <slack-channel>
    notifications.argoproj.io/subscribe.on-sync-failed.slack: <slack-channel>

spec:
  destination:
    namespace: argocd-01            
    server: https://kubernetes.default.svc
  project: argocd-01-config
  source:
    path: argocd-notifications/base
    repoURL: https://github.com/StinkyBenji/notifications-demo
    targetRevision: master
    plugin:
      name: argocd-vault-plugin
      env:
        - name: AVP_K8S_ROLE
          value: vplugin
        - name: AVP_TYPE
          value: vault
        - name: VAULT_ADDR
          value: 'http://vault-server.vault.svc.cluster.local:8200'
        - name: AVP_AUTH_TYPE
          value: k8sCode language: HTML, XML (xml)

Once we deployed the application, we should be able to see a nice topography on the Argo CD web console:

At the same time, you should receive a message on your slack channel about the successful synchronization.

But, what if something naughty happened without us noticing? Don’t worry, this time we will get notified!

Yay! We no longer need to keep checking the status of our Argo CD applications by ourselves. Argo CD notifications will always inform us of “weird stuff” going on in our applications!

What’s next?

Congratulation, we managed to get the notifications from Argo CD. If you are interested in using one-line command to complete the process that we made in the article, you can have a look at these two small demos (openshift-gitops-setup, notifications-demo) I created.