Setting Autoscaler on EKS Using OIDC

So. You want to deploy the Autoscaler on EKS. But, plot twist: you’ve got
OIDC in the mix. How do you do that without summoning an ancient demon from
the depths of AWS documentation?

Well, after a full day of dancing with Terraform dragons and arguing with
variables that behave like goblins, I stumbled across a solution.
(Or maybe it stumbled across me. These things are hard to tell.)

You might think you could just call module.eks.oidc_provider_arn and be
done with it. And in a world governed by logic and kindness, that might
actually work. But our world—our delightful chaos—uses a meta-module.
Which calls other modules. Like so:

  • EKS (the head honcho)
  • GitLab Runner (who shows up late but does the work)
  • Istio (no one really understands it, but it’s important)
  • Etc. (the real MVP)

So, the simple solution? Utterly useless. Because we want to get everything
from var.eks_cluster_name, which we gracefully pass into this meta-monster.
Using the first method would mean hardcoding the cluster name inside the
runner submodule. And hardcoding is, of course, a mortal sin.

Thus, we turn to a solution both elegant and absurdly convoluted. We reach
into the Terraform bag of holding and pull out data. Then we politely ask
aws_iam_openid_connect_provider for its precious arn. We use this in the
trust policy, so our noble runner may indeed be validated by the mighty OIDC.

Still with me? No? Excellent. That means you’re paying attention.

1
2
3
4
5
6
7
8
9
10
11
12
data "aws_eks_cluster" "this" {
  name = "eks-cluster-name"
}

data "aws_iam_openid_connect_provider" "this" {
  url = data.aws_eks_cluster.this.identity[0].oidc[0].issuer
}

# This output is only for debugging purposes, or for staring at while you question your life choices
output "oidc_provider_arn" {
  value = data.aws_iam_openid_connect_provider.this.arn
}

Now, with this magical arn in hand, we can bravely proceed to summon our
aws_iam_role for the Autoscaler:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
resource "aws_iam_role" "autoscaler" {
  name = "gitlab-runner-autoscaler"
  assume_role_policy = templatefile(
    "${path.module}/policies/autoscaler_assumerole.json",
    {
      eks_oidc = data.aws_iam_openid_connect_provider.this.arn
    }
  )
  inline_policy {
    name   = "gitlab-runner-autoscaler-permissions"
    policy = "${path.module}/policies/autoscaler.json"
  }
  tags = var.tags
}

And what sacred bond do we forge in this trust relationship? Behold:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "${eks_oidc_arn}"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "oidc.eks.eu-west-1.amazonaws.com/id/XXX:sub": "system:serviceaccount:devops:*",
                    "oidc.eks.eu-west-1.amazonaws.com/id/XXX:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
  }

And lo! It works. Somehow. Don’t ask why. The code gods have smiled upon you.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# module.gitlabrunner.aws_iam_role.autoscaler will be created
  + resource "aws_iam_role" "autoscaler" {
      + arn                   = (known after apply)
      + assume_role_policy    = jsonencode(
            {
              + Statement = [
                  + {
                      + Action    = "sts:AssumeRoleWithWebIdentity"
                      + Condition = {
                          + StringLike = {
                              + "oidc.eks.eu-west-1.amazonaws.com/id/XXX:aud" = "sts.amazonaws.com"
                              + "oidc.eks.eu-west-1.amazonaws.com/id/XXX:sub" = "system:serviceaccount:devops:*"
                            }
                        }
                      + Effect    = "Allow"
                      + Principal = {
                          + Federated = "arn:aws:iam::5XXX:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/A7CC3E1E7386XXXXX"
                        }
                    },
                ]
              + Version   = "2012-10-17"
            }
        )

So go forth, brave engineer! Your Autoscaler is alive. The OIDC has been
appeased. And you only had to sacrifice a little sanity to get here.

Comments

⬆︎TOP