Skip to content

Commit

Permalink
Airflow worker toleration (bitnami#5925)
Browse files Browse the repository at this point in the history 
* Add strict toleration and affinity settings on worker pods template

* Update chart version and README

* Add the worker.tolerations and worker.affinity checks in statefulset.yaml worker template

Co-authored-by: Tiago Perini <[email protected]>
  • Loading branch information
@tlperini
tlperini and Tiago Perini authored Mar 31, 2021
1 parent 6802bd8 commit 8bae199
Show file tree
Hide file tree
Showing 5 changed files with 35 additions and 5 deletions.
2 changes: 1 addition & 1 deletion bitnami/airflow/Chart.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 32,4 @@ name: airflow
sources:
- https://github.com/bitnami/bitnami-docker-airflow
- https://airflow.apache.org/
version: 8.1.3
version: 8.2.0
4 changes: 4 additions & 0 deletions bitnami/airflow/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 187,8 @@ The following tables lists the configurable parameters of the Airflow chart and

| Parameter | Description | Default |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- |
| `worker.affinity` | Affinity for worker pod assignment. Supersedes the common
affinity configuration | `nil` |
| `worker.args` | Override default container args (useful when using custom images) | `nil` |
| `worker.autoscaling.enabled` | Switch to enable Horizontal Pod Autoscaler for Airflow worker component (only when executor is `CeleryExecutor`). When enable you should also set `resources.requests` | `false` |
| `worker.autoscaling.replicas.max` | Maximum amount of replicas | `3` |
Expand Down Expand Up @@ -234,6 236,8 @@ The following tables lists the configurable parameters of the Airflow chart and
| `worker.resources.requests` | The requested resources for the worker containers | `{}` |
| `worker.rollingUpdatePartition` | Partition update strategy | `nil` |
| `worker.sidecars` | List of sidecar containers to be added to the worker's pods | `nil` |
| `worker.tolerations` | Tolerations for worker pod assignment. Supersedes the common
tolerations configuration | `nil` |
| `worker.updateStrategy` | pdate strategy for the statefulset | `"RollingUpdate"` |

### Airflow database parameters
Expand Down
12 changes: 10 additions & 2 deletions bitnami/airflow/templates/config/configmap.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -169,8 169,12 @@ data:
{{- if .Values.podSecurityContext.enabled }}
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
{{- if .Values.affinity }}
{{- if or .Values.affinity .Values.worker.affinity }}
{{- if .Values.worker.affinity }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.worker.affinity "context" $) | nindent 8 }}
{{- else }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- end }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "worker" "context" $) | nindent 10 }}
Expand All @@ -184,9 188,13 @@ data:
nodeSelector: {{- include "common.tplvalues.render" (dict "value" .Values.worker.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- end }}
{{- if .Values.tolerations }}
{{- if or .Values.tolerations .Values.worker.tolerations }}
{{- if .Values.worker.tolerations }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.worker.tolerations "context" $) | nindent 8 }}
{{- else }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
{{- end }}
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
volumes:
- name: k8s-executor-config
Expand Down
12 changes: 10 additions & 2 deletions bitnami/airflow/templates/worker/statefulset.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 50,12 @@ spec:
securityContext: {{- omit .Values.podSecurityContext "enabled" | toYaml | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "airflow.serviceAccountName" . }}
{{- if .Values.affinity }}
{{- if or .Values.affinity .Values.worker.affinity }}
{{- if .Values.worker.affinity }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.worker.affinity "context" $) | nindent 8 }}
{{- else }}
affinity: {{- include "common.tplvalues.render" (dict "value" .Values.affinity "context" $) | nindent 8 }}
{{- end }}
{{- else }}
affinity:
podAffinity: {{- include "common.affinities.pods" (dict "type" .Values.podAffinityPreset "component" "worker" "context" $) | nindent 10 }}
Expand All @@ -65,9 69,13 @@ spec:
nodeSelector: {{- include "common.tplvalues.render" (dict "value" .Values.worker.nodeSelector "context" $) | nindent 8 }}
{{- end }}
{{- end }}
{{- if .Values.tolerations }}
{{- if or .Values.tolerations .Values.worker.tolerations }}
{{- if .Values.worker.tolerations }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.worker.tolerations "context" $) | nindent 8 }}
{{- else }}
tolerations: {{- include "common.tplvalues.render" (dict "value" .Values.tolerations "context" $) | nindent 8 }}
{{- end }}
{{- end }}
initContainers: {{- include "airflow.git.containers.clone" . | trim | nindent 8 }}
{{- if .Values.worker.initContainers }}
{{- include "common.tplvalues.render" (dict "value" .Values.worker.initContainers "context" $) | trim | nindent 8 }}
Expand Down
10 changes: 10 additions & 0 deletions bitnami/airflow/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -486,6 486,16 @@ worker:
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
## Affinity for worker pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
## Note: this configuration supersedes the global affinity configuration
##
affinity: {}
## Tolerations for worker pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
## Note: this configuration supersedes the global tolerations configuration
##
tolerations: []

## Add labels to all the deployed resources
##
Expand Down

0 comments on commit 8bae199

Please sign in to comment.