The state of play

With the introduction of IAM roles for Service Accounts AWS has made it easy to assign IAM roles to individual pods using the eks.amazonaws.com/role-arn annotation to pod service accounts. However to make this work, the cluster first needs to have an OpenID Connect (OIDC) Provider configured. This configuration could be done manually using the console or via SDK/CLI scripts, but then operational audit, resource ownership, and environment teardown semantics are lost. We want config as code.

This blog post will show you how to configure an OIDC provider for an existing EKS cluster using a single CloudFormation template. It will roll up the multiple command line steps from the AWS guide into CustomResources that will do the configuration for you, and return the URL required to apply to the role.

Other guides exist to show you how this can be done with Terraform and eksctl, so if you are using those tools to manage your infrastructure, you may want to start there.

The repository for the finished template, and the testing resources can be found here: https://github.com/bambooengineering/example-eks-oidc-iam-cloudformation.

Template overview

You can find the full template to download and use in the repo here. It has the following high level structure.

---
Parameters:
  EKSClusterName:
    ... The name of the cluster to configure (not the full arn).

Resources:
  ClusterOIDCURL:
    ... The CustomResource that manages the queries to the cluster for its OIDC URL by using the ClusterOIDCURLFunction. 
        We need this URL to configure the OIDC provider and apply it to namespace constraints in the 
        IAM role.

  ClusterOIDCURLFunction:
    ... The lambda that will query the cluster for the URL.  

  ClusterOIDCProvider:
    ... The CustomResource that will invoke the ClusterOIDCProviderFunction to create the provider.

  ClusterOIDCProviderFunction:
    ... The lambda that actually creates and deletes the provider.

  ClusterOIDCLambdaExecutionRole:
    ... The role that gives the stack sufficient permissions to create the OIDC provider. 
        It is only used during lifecycle operations of this stack. 

Outputs:
  ClusterOIDCURL:
     ... The OpenID Connect URL (without protocol). This must be used later in the pod IAM roles.

Prerequisites

Before you apply the template:

• You will need a running EKS cluster to apply the template to. If you do not have one, you can create one for testing using eksctl with eksctl create cluster.
• Your cluster must be 1.14 or above, and have been upgraded / created after September 2019.
• You absolutely must ensure the aws-cli or aws-sdk version used inside your business logic pods is high enough. See here for a list of versions supported. This is one of the main reasons your pod may simply (and silently) fail to obtain credentials.

The show comes to town

Once you have downloaded the template oidc-provider.yaml, you can configure your cluster with one command:

# Assign your cluster name to an ENV here (the short name, not the full ARN).
# Note, this cluster must already exist. You can create one to test with by using 
# `eksctl create cluster`
# => "curious-badger-1585558460" or similar.
CLUSTER_NAME=<NAME>

# Create the OIDC stack. Make sure the stack and the cluster are in the same region.
aws cloudformation create-stack  \
    --capabilities CAPABILITY_NAMED_IAM \
    --stack-name ${CLUSTER_NAME}-oidc \
    --parameters ParameterKey=EKSClusterName,ParameterValue=${CLUSTER_NAME} \
    --template-body file://oidc-provider.yaml

After a few seconds that stack will create the required resource: an OIDC Provider. You will be able to see it in your AWS console in the IAM section here.

It will also add the cluster OIDC URL to the Outputs of the stack without the "https://" prefix, which makes it easy for applying to any roles you want to then create. You can then wait for it to finish and view the output thus:

# View the OIDC URL:
OIDC_URL=$(aws cloudformation describe-stacks --stack-name ${CLUSTER_NAME}-oidc \
    --query "Stacks[0].Outputs[?OutputKey=='ClusterOIDCURL'].OutputValue" \
    --output text)
echo "OIDC_URL is ${OIDC_URL}"

That's it! Your cluster is now configured. You deserve a beer.

CloudFormation all the way

This guide allows you to configure your EKS clusters to use OIDC, and thus IAM roles, using a single CloudFormation template.

In my next post I will show how pod IAM roles can (and should) be constrained by namespace and service account, again using CloudFormation.

See part 2: Creating EKS IAM roles using CloudFormation.

Good luck on your Kubernetes journey!