The use case

In my previous post I looked at how to configure an OIDC provider for an existing EKS cluster using a single CloudFormation template.

The next step is to use the OIDC URL output of the template to configure IAM roles so that pods are able to assume them. Importantly, those roles should be restricted so they can only be assumed by pods in the intended namespace, and only by the named service account. There are guides available showing how to do this using eksctl, the AWS console or the aws-cli, but what if we aren't using CDK and wanted to keep our code as config, declarative, and entirely in YAML/JSON?

The problem is in the AssumeRolePolicyDocument section. We need to supply a Condition where the key needs to be templated with the name of the cluster. CloudFormation doesn't permit templating keys natively. Fortunately, there is a workaround.

This post will allow you to configure your EKS clusters and create pod IAM roles, all without leaving CloudFormation. It also demonstrates how to create roles that are bound to a set namespace and service account.

The repository for the example resources can be found here: https://github.com/bambooengineering/example-eks-oidc-iam-cloudformation.

Rock and role

The trick is to know that the AssumeRolePolicyDocument value in a template is specified as json. As such, we can perform a !Sub on any value we pass to it, as long as the result is a valid json string. This allows us to template the "key" part of the Condition.

The value part of the Condition applies the Principal of Least Privilege. It states that the pod must be in the namespace test-namespace, and must have the service account named test-service-account.

If a pod attempts to assume this role from another namespace or with any other service account, it will fail.

Here is how to apply the desired constraints to an IAM role using CloudFormation:

Parameters:
  EKSClusterName:
    Type: String
    Description: Name for EKS Cluster

  ClusterOIDCURL:
    Type: String
    Description: The OpenID Connect URL without protocol (the "https://" prefix)

Resources:
  RuntimePodRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "${EKSClusterName}-pod-role"
      # This `AssumeRolePolicyDocument` section states that this business 
      # logic role is permitted to be assumed by pods through the 
      # OpenID Connect provider.
      #
      # We have to drop into a JSON string here as we want to apply 
      # a Condition to restrict the role to pods in the namespace "test-namespace" 
      # and with the service account name "test-service-account". There is no
      # other way to template a StringEquals key in CloudFormation YAML.
      #
      # Use "*" in place of "test-service-account" for all service accounts in 
      # one namespace. It is possible to use "*" in place of 
      # "test-namespace:test-service-account" to permit all pods in a cluster 
      # to assume the role, but don't do this unless the permission is harmless
      # or genuinely needed globally.
      AssumeRolePolicyDocument: !Sub |
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Federated": "arn:aws:iam::${AWS::AccountId}:oidc-provider/${ClusterOIDCURL}"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "${ClusterOIDCURL}:sub": "system:serviceaccount:test-namespace:test-service-account"
                }
              }
            }
          ]
        }
      Policies:
      - PolicyName: example-access-policy
        PolicyDocument:
          # These statements are the example business logic permissions required by this pod
          Statement:
          - Effect: Allow
            Action:
            - sns:GetTopicAttributes
            Resource: "*"
      Path: "/"

The same technique is possible with JSON templates, though much harder to read as the JSON needs to be escaped. Here is the Role part of a JSON template:

  {
    "RuntimePodRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": {
          "Fn::Sub": "${EKSClusterName}-pod-role"
        },
        "AssumeRolePolicyDocument": {
          "Fn::Sub": "{\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Federated\":\"arn:aws:iam::${AWS::AccountId}:oidc-provider/${ClusterOIDCURL}\"},\"Action\":\"sts:AssumeRoleWithWebIdentity\",\"Condition\":{\"StringEquals\":{\"${ClusterOIDCURL}:sub\":\"system:serviceaccount:test-namespace:test-service-account\"}}}],\"Version\":\"2012-10-17\"}"
        },
        "Policies": ["Policies here..."],
        "Path": "/"
      }
    }
  }

Role your own

We can now demonstrate the cluster will configure service accounts correctly by performing a test. Clone the example repo and change to that directory.

git clone https://github.com/bambooengineering/example-eks-oidc-iam-cloudformation.git
cd example-eks-oidc-iam-cloudformation

You now have access to the two test files:

• oidc-test-cloudformation.yaml. A CloudFormation template that creates a test SNS topic and IAM role that is allowed to describe that topic.
• oidc-test-kubernetes.yaml. A Kubernetes template that creates a service account, and a pod that uses that account.

Now use the CloudFormation file to set up a test role, and prove the json templating system described above:

# Set the OIDC URL. If you used the template in my previous post, it is in 
# the stack output. It is of the form:
# "oidc.eks.eu-west-1.amazonaws.com/id/896A662285A271779972DA40B3D46EA8"
OIDC_URL=<Your URL>
echo "OIDC_URL is ${OIDC_URL}"

# Create the test elements
aws cloudformation create-stack  \
    --capabilities CAPABILITY_NAMED_IAM \
    --stack-name ${CLUSTER_NAME}-oidc-test \
    --parameters ParameterKey=EKSClusterName,ParameterValue=${CLUSTER_NAME} ParameterKey=ClusterOIDCURL,ParameterValue=${OIDC_URL} \
    --template-body file://oidc-test-cloudformation.yaml

# Use a waiter to wait for that stack to complete
aws cloudformation wait stack-create-complete --stack-name ${CLUSTER_NAME}-oidc-test

# Get the test role arn 
TEST_ROLE_ARN=$(aws cloudformation describe-stacks --stack-name ${CLUSTER_NAME}-oidc-test \
    --query "Stacks[0].Outputs[?OutputKey=='RuntimePodRoleArn'].OutputValue" \
    --output text)
echo "TEST_ROLE_ARN is ${TEST_ROLE_ARN}"

We now have the test IAM role, preconfigured to be assumable by a pod in this cluster.

Using the second file we use kubectl to launch a pod in the cluster, and execute a command in it to describe the test SNS topic (as permitted by the test role we have created). Note, the pod has been assigned a service account that has the role annotated with eks.amazonaws.com/role-arn:

# Create a service account and test pod with a kubernetes resource file, templating in some values.

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
sed -e "s/AWS_ACCOUNT_ID/$AWS_ACCOUNT_ID/g" -e "s/CLUSTER_NAME/$CLUSTER_NAME/g" \
    oidc-test-kubernetes.yaml | kubectl apply -f -

# The above will have launched one example pod that we can use to test our role.
# Once the pod is running (may take 30s), let's find the pod's name.
POD=$(kubectl get pod -l app=test-deployment -o jsonpath="{.items[0].metadata.name}" \
    --namespace test-namespace)
echo "Found pod ${POD}"

# The acid test. Once the pod is running, use it to try to describe the example SNS topic.
# This command will use kubectl to invoke a command inside that remote pod. 
# You will see the SNS `Attributes` being written out.
# Make sure you have AWS_DEFAULT_REGION set
kubectl exec -ti $POD --namespace test-namespace -- \
    aws sns get-topic-attributes \
    --region ${AWS_DEFAULT_REGION} \
    --topic-arn arn:aws:sns:${AWS_DEFAULT_REGION}:${AWS_ACCOUNT_ID}:${CLUSTER_NAME}-example-topic

# The above will have out put the SNS topic attributes, as seen by the pod. Success!

# To clean up:
kubectl delete -f oidc-test-kubernetes.yaml
aws cloudformation delete-stack --stack-name ${CLUSTER_NAME}-oidc-test

Your work here is done.

Little fluffy clouds

By keeping your configuration entirely in CloudFormation templates you can maintain a cleaner and simpler deploy pipeline, without sacrificing the necessary constraints on role usage by service account and namespace. Your boss will be pleased.

Good luck on your Kubernetes journey!