Using GitOps with Linkerd with Argo CD

GitOps is an approach to automate the management and delivery of your Kubernetes infrastructure and applications using Git as a single source of truth. It usually utilizes some software agents to detect and reconcile any divergence between version-controlled artifacts in Git with what’s running in a cluster.

This guide will show you how to set up Argo CD to manage the installation and upgrade of Linkerd using a GitOps workflow.

Specifically, this guide provides instructions on how to securely generate and manage Linkerd’s mTLS private keys and certificates using Sealed Secrets and cert-manager. It will also show you how to integrate the auto proxy injection feature into your workflow. Finally, this guide conclude with steps to upgrade Linkerd to a newer version following a GitOps workflow.

The software and tools used in this guide are selected for demonstration purposes only. Feel free to choose others that are most suited for your requirements.

You will need to clone this example repository to your local machine and replicate it in your Kubernetes cluster following the steps defined in the next section.

This guide uses the step cli to create certificates used by the Linkerd clusters to enforce mTLS, so make sure you have installed step for your environment.

Set up the repositories

Clone the example repository to your local machine:

git clone https://github.com/linkerd/linkerd-examples.git

This repository will be used to demonstrate Git operations like add, commit and push later in this guide.

Add a new remote endpoint to the repository to point to the in-cluster Git server, which will be set up in the next section:

cd linkerd-examples

git remote add git-server git://localhost/linkerd-examples.git

Deploy the Git server to the scm namespace in your cluster:

kubectl apply -f gitops/resources/git-server.yaml

Later in this guide, Argo CD will be configured to watch the repositories hosted by this Git server.

Confirm that the Git server is healthy:

kubectl -n scm rollout status deploy/git-server

Clone the example repository to your in-cluster Git server:

git_server=`kubectl -n scm get po -l app=git-server -oname | awk -F/ '{ print $2 }'`

kubectl -n scm exec "${git_server}" -- \
  git clone --bare https://github.com/linkerd/linkerd-examples.git

Confirm that the remote repository is successfully cloned:

kubectl -n scm exec "${git_server}" -- ls -al /git/linkerd-examples.git

Confirm that you can push from the local repository to the remote repository via port-forwarding:

kubectl -n scm port-forward "${git_server}" 9418  &

git push git-server master

Install the Argo CD CLI

Before proceeding, install the Argo CD CLI in your local machine by following the instructions relevant to your OS.

Deploy Argo CD

Install Argo CD:

kubectl create ns argocd

kubectl -n argocd apply -f \
  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Confirm that all the pods are ready:

for deploy in "dex-server" "redis" "repo-server" "server"; \
  do kubectl -n argocd rollout status deploy/argocd-${deploy}; \
done

kubectl -n argocd rollout status statefulset/argocd-application-controller

Use port-forward to access the Argo CD dashboard:

kubectl -n argocd port-forward svc/argocd-server 8080:443  \
  > /dev/null 2>&1 &

The Argo CD dashboard is now accessible at https://localhost:8080, using the default admin username and password.

Authenticate the Argo CD CLI:

password=`kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d`

argocd login 127.0.0.1:8080 \
  --username=admin \
  --password="${password}" \
  --insecure

Configure project access and permissions

Set up the demo project to group our applications:

kubectl apply -f gitops/project.yaml

This project defines the list of permitted resource kinds and target clusters that our applications can work with.

Confirm that the project is deployed correctly:

argocd proj get demo

On the dashboard:

Deploy the applications

Deploy the main application which serves as the “parent” for all the other applications:

kubectl apply -f gitops/main.yaml

Confirm that the main application is deployed successfully:

argocd app get main

Sync the main application:

argocd app sync main

Notice that only the main application is synchronized.

Next, we will synchronize the remaining applications individually.

Deploy cert-manager

Synchronize the cert-manager application:

argocd app sync cert-manager

Confirm that cert-manager is running:

for deploy in "cert-manager" "cert-manager-cainjector" "cert-manager-webhook"; \
  do kubectl -n cert-manager rollout status deploy/${deploy}; \
done

Deploy Sealed Secrets

Synchronize the sealed-secrets application:

argocd app sync sealed-secrets

Confirm that sealed-secrets is running:

kubectl -n kube-system rollout status deploy/sealed-secrets

Create mTLS trust anchor

Before proceeding with deploying Linkerd, we will need to create the mTLS trust anchor. Then we will also set up the linkerd-bootstrap application to manage the trust anchor certificate.

Create a new mTLS trust anchor private key and certificate:

step certificate create root.linkerd.cluster.local sample-trust.crt sample-trust.key \
  --profile root-ca \
  --no-password \
  --not-after 43800h \
  --insecure

Confirm the details (encryption algorithm, expiry date, SAN etc.) of the new trust anchor:

step certificate inspect sample-trust.crt

Before creating the SealedSecret, make sure you have installed the kubeseal utility, as instructed here

Now create the SealedSecret resource to store the encrypted trust anchor:

kubectl create ns linkerd
kubectl -n linkerd create secret tls linkerd-trust-anchor \
  --cert sample-trust.crt \
  --key sample-trust.key \
  --dry-run=client -oyaml | \
kubeseal --controller-name=sealed-secrets -oyaml - | \
kubectl patch -f - \
  -p '{"spec": {"template": {"type":"kubernetes.io/tls", "metadata": {"labels": {"linkerd.io/control-plane-component":"identity", "linkerd.io/control-plane-ns":"linkerd"}, "annotations": {"linkerd.io/created-by":"linkerd/cli stable-2.12.0"}}}}}' \
  --dry-run=client \
  --type=merge \
  --local -oyaml > gitops/resources/linkerd/trust-anchor.yaml

This will overwrite the existing SealedSecret resource in your local gitops/resources/linkerd/trust-anchor.yaml file. We will push this change to the in-cluster Git server.

Confirm that only the spec.encryptedData is changed:

git diff gitops/resources/linkerd/trust-anchor.yaml

Commit and push the new trust anchor secret to your in-cluster Git server:

git add gitops/resources/linkerd/trust-anchor.yaml

git commit -m "update encrypted trust anchor"

git push git-server master

Confirm the commit is successfully pushed:

kubectl -n scm exec "${git_server}" -- git --git-dir linkerd-examples.git log -1

Deploy linkerd-bootstrap

Synchronize the linkerd-bootstrap application:

argocd app sync linkerd-bootstrap

SealedSecrets should have created a secret containing the decrypted trust anchor. Retrieve the decrypted trust anchor from the secret:

trust_anchor=`kubectl -n linkerd get secret linkerd-trust-anchor -ojsonpath="{.data['tls\.crt']}" | base64 -d -w 0 -`

Confirm that it matches the decrypted trust anchor certificate you created earlier in your local sample-trust.crt file:

diff -b \
  <(echo "${trust_anchor}" | step certificate inspect -) \
  <(step certificate inspect sample-trust.crt)

Deploy Linkerd

Now we are ready to install Linkerd. The decrypted trust anchor we just retrieved will be passed to the installation process using the identityTrustAnchorsPEM parameter.

Prior to installing Linkerd, note that the identityTrustAnchorsPEM parameter is set to an “empty” certificate string:

argocd app get linkerd-control-plane -ojson | \
  jq -r '.spec.source.helm.parameters[] | select(.name == "identityTrustAnchorsPEM") | .value'

We will override this parameter in the linkerd application with the value of ${trust_anchor}.

Locate the identityTrustAnchorsPEM variable in your local gitops/argo-apps/linkerd-control-plane.yaml file, and set its value to that of ${trust_anchor}.

Ensure that the multi-line string is indented correctly. E.g.,

  source:
    chart: linkerd-control-plane
    repoURL: https://helm.linkerd.io/stable
    targetRevision: 1.9.0
    helm:
      parameters:
      - name: identityTrustAnchorsPEM
        value: |
          -----BEGIN CERTIFICATE-----
          MIIBlTCCATygAwIBAgIRAKQr9ASqULvXDeyWpY1LJUQwCgYIKoZIzj0EAwIwKTEn
          MCUGA1UEAxMeaWRlbnRpdHkubGlua2VyZC5jbHVzdGVyLmxvY2FsMB4XDTIwMDkx
          ODIwMTAxMFoXDTI1MDkxNzIwMTAxMFowKTEnMCUGA1UEAxMeaWRlbnRpdHkubGlu
          a2VyZC5jbHVzdGVyLmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE PUp
          IR74PsU geheoyseycyquYyes5eeksIb5FDm8ptOXQ2xPcBpvesZkj6uIyS3k4qV
          E0S9VtMmHNeycL7446NFMEMwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYB
          Af8CAQEwHQYDVR0OBBYEFHypCh7hiSLNxsKhMylQgqD9t7NNMAoGCCqGSM49BAMC
          A0cAMEQCIEWhI86bXWEd4wKTnG07hBfBuVCT0bxopaYnn3wRFx7UAiAwXyh5uaVg
          MwCC5xL PM bm3PRqtrmI6TocWH07GbMxg==
          -----END CERTIFICATE-----          

Confirm that only one spec.source.helm.parameters.value field is changed:

git diff gitops/argo-apps/linkerd-control-plane.yaml

Commit and push the changes to the Git server:

git add gitops/argo-apps/linkerd-control-plane.yaml

git commit -m "set identityTrustAnchorsPEM parameter"

git push git-server master

Synchronize the main application:

argocd app sync main

Confirm that the new trust anchor is picked up by the linkerd application:

argocd app get linkerd-control-plane -ojson | \
  jq -r '.spec.source.helm.parameters[] | select(.name == "identityTrustAnchorsPEM") | .value'

Synchronize the linkerd-crds and linkerd-control-plane applications:

argocd app sync linkerd-crds
argocd app sync linkerd-control-plane

Check that Linkerd is ready:

linkerd check

Test with emojivoto

Deploy emojivoto to test auto proxy injection:

argocd app sync emojivoto

Check that the applications are healthy:

for deploy in "emoji" "vote-bot" "voting" "web" ; \
  do kubectl -n emojivoto rollout status deploy/${deploy}; \
done

Upgrade Linkerd to 2.12.1

(Assuming 2.12.1 has already been released ;-) )

Use your editor to change the spec.source.targetRevision field to 1.9.3 (that’s the Helm chart version corresponding to linkerd stable-2.12.1) in the gitops/argo-apps/linkerd-control-plane.yaml file:

Confirm that only the targetRevision field is changed:

git diff gitops/argo-apps/linkerd-control-plane.yaml

Commit and push this change to the Git server:

git add gitops/argo-apps/linkerd-control-plane.yaml

git commit -m "upgrade Linkerd to 2.12.1"

git push git-server master

Synchronize the main application:

argocd app sync main

Synchronize the linkerd-control-plane application:

argocd app sync linkerd-control-plane

Confirm that the upgrade completed successfully:

linkerd check

Confirm the new version of the control plane:

linkerd version

Clean up

All the applications can be removed by removing the main application:

argocd app delete main --cascade=true