Using !Reference Your Gitlab Pipelines

How to resuse pipelines code using Reference

After some time, as we returned to recreate the entire set of pipelines we have,
we discovered this function that will save us a lot of time.

Take a look at this job we have:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
deploy-lz:
  extends: .deploy-lz-standard
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: [REDACTED]
  variables:
    <<: *common-vars
    AWS_REGION: eu-west-2
    NAMESPACE: monitoring
    HELM_VALUES: "--values k8s/lz/values.yml ${HELM_METADATA_VALUES}"
    HELM_EXT: "
      --set deployments.app.image=$AWS_ECR/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
      --set deployments.app.environment.VAULT_SMOKE_APP=${CI_RUNNER_TAGS}
      --set deployments.app-istio.image=$AWS_ECR/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA
      --set deployments.app-istio.environment.VAULT_SMOKE_APP=${CI_RUNNER_TAGS}
      "

This is how we are starting to deploy using the pipelines. As you can see, the
job is very small, and it also leaves little room for anyone to corrupt it.

But we have many things in the background that make it so that a developer only
needs to specify this job, and it’s GitLab itself that decides:

  • Where it is deployed
  • In which cluster
  • And with what permissions

Now, let’s get to the important part: !Reference. This job is a matryoshka of
jobs; it calls this job:

1
2
3
4
5
6
7
8
9
10
11
12
.deploy-lz-standard: &deploy-lz-standard
  stage: deploy
  image: $AWS_ECR/gitlab-ci/deployer:v1.0.0
  tags:
    - $RUNNER_TAGS
  before_script:
    - !reference [.oidc_authenticate, before_script]
  script:
    - !reference [.default_helm_deploy, script]
  when: manual
  needs:
    - build-lz

And here’s where the magic of !Reference comes in. We used to use the famous
<<: *, but it’s a hassle and doesn’t merge arrays; it just overwrites.
With !Reference, we can specify where we want to use those snippets,
allowing anyone to reuse them wherever they wish.

Continuing with the example, let’s look at those two snippets we referenced:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
.deployer_prepare:
  before_script:
    # Package installation
    - apt-get update && apt-get install -y --no-install-recommends curl tar bash jq python3 python3-pip git gcc build-essential && apt-get clean && rm -rf /var/lib/apt/lists/*
    # HashiCorp packages cleanup
    - |
      if [ -f /etc/apt/sources.list.d/hashicorp.list ]; then
        echo "Cleaning up hashicorp.list..."
        rm /etc/apt/sources.list.d/hashicorp.list
      fi
      if [ -f /etc/apt/sources.list.d/archive_uri-https_apt_releases_hashicorp_com-bookworm.list ]; then
        echo "Cleaned up archive_uri-https_apt_releases_hashicorp_com-bookworm.list..."
        rm /etc/apt/sources.list.d/archive_uri-https_apt_releases_hashicorp_com-bookworm.list
      fi

.oidc_authenticate:
  before_script:
    - !reference [.deployer_prepare, before_script]
    # Assume role using GitLab OIDC
    - pip install awscli --no-cache-dir --break-system-packages
    - echo "$GITLAB_OIDC_TOKEN" > /tmp/gitlab-id-token
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/gitlab-id-token
    - |
      CREDS=$(aws sts assume-role-with-web-identity \
        --role-arn "$(aws iam get-role --role-name $AWS_DEFAULT_GITLAB_RUNNER_ROLE --query 'Role.Arn' --output text)" \
        --role-session-name "gitlab-ci-session" \
        --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
        --duration-seconds 3600)

      export AWS_ACCESS_KEY_ID=$(echo "$CREDS" | jq -r .Credentials.AccessKeyId)
      export AWS_SECRET_ACCESS_KEY=$(echo "$CREDS" | jq -r .Credentials.SecretAccessKey)
      export AWS_SESSION_TOKEN=$(echo "$CREDS" | jq -r .Credentials.SessionToken)
    - mkdir -p ~/.aws
    - aws eks
      --region $AWS_REGION
      update-kubeconfig
      --name $KUBE_CLUSTER
      --alias $KUBE_CLUSTER
    - |
      for repo in $(echo $HELM_REPOS | sed "s/,/ /g")
      do
        helm repo add $(echo $repo | sed "s/;/ /g")
      done
    - "helm repo update || :"

That’s where the magic lies: we have 3 levels of nested jobs called from
wherever we want, so if someone wants to deploy, they only have to call the
first job. And if we want to change something in the authentication, we only
need to modify it in .oidc_authenticate, and that change will propagate.

This is code reuse, which (in addition to increasing the blast radius) allows
developers to abstract away a lot of code that they neither want nor need to know.

I hope this serves as inspiration!

Comments

26-09-2025
⬆︎TOP