Guard pipeline jobs from multiple simultaneous executions
$ PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard -w 3600
$ my-unguarded-deployment-task --to=production
gitlab-job-guard
will block if it detects other pipelines running for
the current project to avoid multiple pipelines from clobbering up a
deployment/environment.
While gitlab will auto-cancel redundant, pending pipelines for the same branch
by default - this is not the case for multiple pipelines from different
branches targeting a particular deployment/environment. Gitlab has no way to
detect or control these user-defined branch-to-environment mappings and this
means environments can easily be left in an unsafe/broken state. (e.g.
terraform apply
or ansible
, etc from different pipelines running at the same
time).
gitlab-job-guard
uses the Gitlab API to determine if existing pipelines are
scheduled and to backoff-and-retry until it is safe to proceed. Conflicts
are detected by user-defined matches on pipeline ref names (branch, tag, etc)
and/or pipeline status.
The simplest usage would likely be placing gitlab-job-guard
in a
before_script
section in your gitlab-ci.yml
to protect all jobs (though this
can slow things down).
before_script:
- PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard
Though often, this is only needed to guard jobs that share common state/data (i.e. a deployment environment, an artifact build/release, etc).
deploy-production:
stage: deploy
script:
- PRIVATE_TOKEN="$GITLAB_API_TOKEN" gitlab-job-guard
- my-unguarded-deployment-task --to=production
or to guard something like a terraform
job running for tags.
provision-infrastructure:
stage: provision
script:
- export PRIVATE_TOKEN="$GITLAB_API_TOKEN"
- gitlab-job-guard --guard-ref-regex='^v[0-9\.] ' # Regex matches tags
- terraform plan ...
- terraform apply ...
only:
- tags
To hold jobs for a collisions on pattern matches on the ref/branch name.
gitlab-job-guard -c=^master$ # Match branch names matching 'master' exactly
gitlab-job-guard -c=^(master|dev(elop)?)$ # Match any of the mainline branches
gitlab-job-guard -c=^(feature|release|hotfix)/ # Match any gitflow transient branch prefixes
gitlab-job-guard -c=^[0-9]\- # Match branch names beginning with a number
# and dash ignoring all other text.
# e.g. a gitlab branch made from an issue
gitlab-job-guard -c=^v?[\d.] $ # Match (semver) tags like v1.0.9, 2.0
gitlab-job-guard -c=^environment/ # Match any environment deployments?
gitlab-job-guard -c=^environment/dc1. # Match environment deployments to DC1?
gitlab-job-guard -c="$CI_BUILD_REF_NAME" # Match current branch name (partially).
# i.e. 'master' matches 'feature/master-document'
gitlab-job-guard -c="^$CI_BUILD_REF_NAME$" # Match current branch name (exactly).
# i.e. 'master' does not match 'master-deployment'
gitlab-job-guard -c='. ' -s='running|pending' # Match any pipeline in running or pending state
To hold a job for a collision on part of the ref name (e.g. on branch prefix
such as feature/
or hotfix/
or release/
, etc a la gitflow
).
# Assuming CI_BUILD_REF_NAME=feature/foo
CI_BUILD_REF_PREFIX=$(echo "$CI_BUILD_REF_NAME" | sed -r 's@(. /)(. )@\1@')
# CI_BUILD_REF_PREFIX now contains 'feature/'
gitlab-job-guard -c="^$CI_BUILD_REF_PREFIX" -s='running|pending'
For long pipelines, this solution can have subtle consequences with growing queues and increased contention and unpredictability as to which pipeline is the first-past-the-post. An older pipeline taking precedence over newer commits if often not desired and newer pipelines always winning is probably desired.
- Handle existing conflicting pipelines - cancel them or give-way.
- Narrow down conflicts to jobs (
CI_JOB_NAME
) or stages (CI_JOB_STAGE
) so that other parts of the pipelines that do not share state are allowed to run freely.