Apache Airflow is a platform to programmatically author, schedule and monitor workflows.

$ helm install my-release bitnami/airflow

This chart bootstraps an Apache Airflow deployment on a Kubernetes cluster using the Helm package manager.

Bitnami charts can be used with Kubeapps for deployment and management of Helm Charts in clusters. This Helm chart has been tested on top of Bitnami Kubernetes Production Runtime (BKPR). Deploy BKPR to get automated TLS certificates, logging and monitoring for your applications.

• Kubernetes 1.12
• Helm 3.1.0

To install the chart with the release name my-release:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm install my-release bitnami/airflow

These commands deploy Airflow on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

To uninstall/delete the my-release deployment:

$ helm delete my-release

The command removes all the Kubernetes components associated with the chart and deletes the release.

The following tables lists the configurable parameters of the Airflow chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

$ helm install my-release \
               --set auth.username=my-user \
               --set auth.password=my-passsword \
               --set auth.fernetKey=my-fernet-key \
               bitnami/airflow

The above command sets the credentials to access the Airflow web UI.

NOTE: Once this chart is deployed, it is not possible to change the application's access credentials, such as usernames or passwords, using Helm. To change these application credentials after deployment, delete any persistent volumes (PVs) used by the chart and re-deploy it, or use the application's built-in administrative tools if available.

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

$ helm install my-release -f values.yaml bitnami/airflow

Tip: You can use the default values.yaml

It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.

Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.

A Fernet key is required in order to encrypt password within connections. The Fernet key must be a base64-encoded 32-byte key.

Learn how to generate one here

There are two different ways to load your custom DAG files into the Airflow chart. All of them are compatible so you can use more than one at the same time.

You can manually create a config map containing all your DAG files and then pass the name when deploying Airflow chart. For that, you can pass the option dagsConfigMap.

You can store all your DAG files on GitHub repositories and then clone to the Airflow pods with an initContainer. The repositories will be periodically updated using a sidecar container. In order to do that, you can deploy airflow with the following options:

NOTE: When enabling git synchronization, an init container and sidecar container will be added for all the pods running airflow, this will allow scheduler, worker and web component to reach dags if it was needed.

git.dags.enabled=true
git.dags.repositories[0].repository=https://github.com/USERNAME/REPOSITORY
git.dags.repositories[0].name=REPO-IDENTIFIER
git.dags.repositories[0].branch=master

If you use a private repository from GitHub, a possible option to clone the files is using a Personal Access Token and using it as part of the URL: https://USERNAME:[email protected]/USERNAME/REPOSITORY

You can load plugins into the chart by specifying a git repository containing the plugin files. The repository will be periodically updated using a sidecar container. In order to do that, you can deploy airflow with the following options:

NOTE: When enabling git synchronization, an init container and sidecar container will be added for all the pods running airflow, this will allow scheduler, worker and web component to reach plugins if it was needed.

git.plugins.enabled=true
git.plugins.repositories[0].repository=https://github.com/teamclairvoyant/airflow-rest-api-plugin.git
git.plugins.repositories[0].branch=v1.0.9-branch
git.plugins.repositories[0].path=plugins

You can use an existing secret to configure your Airflow auth, external Postgres, and extern Redis^TM passwords:

postgresql.enabled=false
externalDatabase.host=my.external.postgres.host
externalDatabase.user=bn_airflow
externalDatabase.database=bitnami_airflow
externalDatabase.existingSecret=all-my-secrets

redis.enabled=false
externalRedis.host=my.external.redis.host
externalRedis.existingSecret=all-my-secrets

auth.existingSecret=all-my-secrets

The expected secret resource looks as follows:

apiVersion: v1
kind: Secret
metadata:
  name: all-my-secrets
type: Opaque
data:
  airflow-password: "Smo1QTJLdGxXMg=="
  airflow-fernetKey: "YVRZeVJVWnlXbU4wY1dOalVrdE1SV3cxWWtKeFIzWkVRVTVrVjNaTFR6WT0="
  postgresql-password: "cG9zdGdyZXMK"
  redis-password: "cmVkaXMK"

This is useful if you plan on using Bitnami's sealed secrets to manage your passwords.

This chart allows you to set your custom affinity using the affinity parameter. Find more information about Pod's affinity in the kubernetes documentation.

As an alternative, you can use of the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common chart. To do so, set the podAffinityPreset, podAntiAffinityPreset, or nodeAffinityPreset parameters.

This chart allows you to mount volumes using extraVolumes and extraVolumeMounts in all 3 airflow components (web, scheduler, worker). Mounting a requirements.txt using these options to /bitnami/python/requirements.txt will execute pip install -r /bitnami/python/requirements.txt on container start. Reference

This chart allows you to set network policies that will rectrict the access to the deployed pods in the cluster. Basically, no other pods apart from Scheduler's pods may access Worker's pods and no other pods apart from Web's pods may access Worker's ones. To do so, set networkPolicies.enabled=true.

Airflow supports different executors runtimes and this chart provides support for the following ones.

Celery executor is the default value for this chart with it you can scale out the number of workers. To point the executor parameter to CeleryExecutor you need to do something, you just install the chart with default parameters.

The kubernetes executor is introduced in Apache Airflow 1.10.0. The Kubernetes executor will create a new pod for every task instance using the pod_template.yaml that you can find templates/config/configmap.yaml, otherwise you can override this template using worker.podTemplate. To enable KubernetesExecutor set the following parameters.

NOTE: Redis^TM is not needed to be deployed when using KubernetesExecutor so you must disable it using redis.enabled=false.

executor=KubernetesExecutor
redis.enabled=false
rbac.create=true
serviceaccount.create=true

Local executor runs tasks by spawning processes in the Scheduler pods. To enable LocalExecutor set the following parameters.

executor=LocalExecutor
redis.enabled=false

This executor will only run one task instance at a time in the Scheduler pods. For production use case, please use other executors. To enable SequentialExecutor set the following parameters.

executor=SequentialExecutor
redis.enabled=false

Sometime when using large workloads a fixed number of worker pods may make task to take a long time to be executed. This chart provide two ways for scaling worker pods.

• If you are using KubernetesExecutor auto scaling pods would be done by the Scheduler without adding anything more.
• If you are using SequentialExecutor you would have to enable worker.autoscaling to do so, please, set the following parameters. It will use autoscaling by default configuration that you can change using worker.autoscaling.replicas.* and worker.autoscaling.targets.*.

worker.autoscaling.enabled=true
worker.resources.requests.cpu=200m
worker.resources.requests.memory=250Mi

The Bitnami Airflow chart relies on the PostgreSQL chart persistence. This means that Airflow does not persist anything.

Find more information about how to deal with common errors related to Bitnami’s Helm charts in this troubleshooting guide.

On November 13, 2020, Helm v2 support was formally finished, this major version is the result of the required changes applied to the Helm Chart to be able to incorporate the different features added in Helm v3 and to be consistent with the Helm project itself regarding the Helm v2 EOL.

• Previous versions of this Helm Chart use apiVersion: v1 (installable by both Helm 2 and 3), this Helm Chart was updated to apiVersion: v2 (installable by Helm 3 only). Here you can find more information about the apiVersion field.

• Move dependency information from the requirements.yaml to the Chart.yaml.

• After running helm dependency update, a Chart.lock file is generated containing the same structure used in the previous requirements.lock.

• The different fields present in the Chart.yaml file has been ordered alphabetically in a homogeneous way for all the Bitnami Helm Charts.

• Several parameters were renamed or disappeared in favor of new ones on this major version:

  • The image objects have been moved to its corresponding component object, e.g: workerImage.* now is located at worker.image.*.
  • The prefix airflow has been removed. Therefore, parameters prefixed with airflow are now at root level, e.g. airflow.loadExamples now is loadExamples or airflow.worker.resources now is worker.resources.
  • Parameters related to the git features has completely been refactored, please see how to configure git for dags and plugins sections for more details.
    • They have been moved to git.* prefix.
    • airflow.cloneDagsFromGit.* no longer exists, instead you must use git.dags.* and git.dags.repositories[*] has been introduced that will add support for multiple repositories.
    • airflow.clonePluginsFromGit.* no longer exists, instead you must use git.plugins.*. airflow.clonePluginsFromGit.repository, airflow.clonePluginsFromGit.branch and airflow.clonePluginsFromGit.path have been removed in favour of git.dags.repositories[*].*.
  • Liveness and rediness probe have been separated by components airflow.livenessProbe.* and airflow.redinessProbe have been removed in favour of web.livenessProbe, worker.livenessProbe, web.redinessProbe and worker.redinessProbe.
  • airflow.baseUrl has been moved to web.baseUrl.
  • Security context has been migrated to the bitnami standard way so that securityContext.* has been divided into podSecurityContext.* that will define the fsGroup for all the containers in the pod and containerSecurityContext.* that will define the user id that will run the main containers.
  • Both bitnami/postgresql and bitnami/redis have been upgraded to their latest major versions, 10.x.x and 11.x.x respectively, find more info in their READMEs bitnami/postgresql and bitnami/redis
  • ./files/dags/*.py will not be include in the deployment any more.

• Some new features:

  • The folder structure has changed, we separated yaml manifest into folders by components.
  • Network policies has been added see more in the Enabling network policies section.
  • Executors, this new version add support for the different executors that airflow implements, see more in the Executors section.
  • Worker scaling functionality has been added, see more in the Scaling worker pods.
  • Pod disruption budget has been added.

• If you want to upgrade to this version using Helm v2, this scenario is not supported as this version doesn't support Helm v2 anymore
• If you installed the previous version with Helm v2 and wants to upgrade to this version with Helm v3, please refer to the official Helm documentation about migrating from Helm v2 to v3
• If you want to upgrade to this version from a previous one installed with Helm v3, you can try to follow the following steps:

NOTE: Please, create a backup of your database before running any of those actions.

Having an already existing chart release called airflow and deployed like

$ helm install airflow bitnami/airflow \
    --set airflow.loadExamples=true \
    --set airflow.baseUrl=http://127.0.0.1:8080

$ export AIRFLOW_PASSWORD=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-password}" | base64 --decode)
$ export AIRFLOW_FERNETKEY=$(kubectl get secret --namespace default airflow -o jsonpath="{.data.airflow-fernetKey}" | base64 --decode)
$ export POSTGRESQL_PASSWORD=$(kubectl get secret --namespace default airflow-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
$ export REDIS_PASSWORD=$(kubectl get secret --namespace default airflow-redis -o jsonpath="{.data.redis-password}" | base64 --decode)
$ export POSTGRESQL_PVC=$(kubectl get pvc -l app.kubernetes.io/instance=postgresql,role=master -o jsonpath="{.items[0].metadata.name}")

Once the new version change fields in the statefulset that will make the upgrade action to fail you will need to remove them

NOTE: Don't forget to set --cascade=false

$ kubectl delete statefulsets.apps --cascade=false airflow-postgresql
$ kubectl delete statefulsets.apps --cascade=false airflow-worker

NOTE: Please remember to migrate all the values to its new path following the above notes, e.g: airflow.loadExamples -> loadExamples or airflow.baseUrl=http://127.0.0.1:8080 -> web.baseUrl=http://127.0.0.1:8080.

$ helm upgrade airflow bitnami/airflow \
    --set loadExamples=true \
    --set web.baseUrl=http://127.0.0.1:8080 \
    --set auth.password=$AIRFLOW_PASSWORD \
    --set auth.fernetKey=$AIRFLOW_FERNETKEY \
    --set postgresql.postgresqlPassword=$POSTGRESQL_PASSWORD \
    --set postgresql.persistence.existingClaim=$POSTGRESQL_PVC \
    --set redis.password=$REDIS_PASSWORD \
    --set redis.cluster.enabled=true

$ kubectl delete pod airflow-postgresql-0

• https://docs.bitnami.com/tutorials/resolve-helm2-helm3-post-migration-issues/
• https://helm.sh/docs/topics/v2_v3_migration/
• https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/

This version also introduces bitnami/common, a library chart as a dependency. More documentation about this new utility could be found here. Please, make sure that you have updated the chart dependencies before executing any upgrade.

This release adds support for LDAP authentication.

This release updates the PostgreSQL chart dependency to use PostgreSQL 11.x. You need to migrate the existing PostgreSQL data to this version before upgrading to this release. For more information follow this link.