Introduction

Accessing AWS resources from outside AWS using only temporary credentials and without creating an IAM user with an access key is a problem that everyone who frequently works with AWS faces at one point. AWS IAM Roles Anywhere helps to solve this issue by enabling applications running on servers on premises to access AWS resources with AWS temporary credentials. It relies on public key infrastructure (PKI) for establishing trust between AWS and the workloads having certificates issued by your own private certificate authority (CA). You create a trust anchor by referencing your own certificate authority (CA) and then configure a role to trust IAM Roles Anywhere by mentioning the trust anchor in the trust policy. In addition, you place a condition in the trust policy to restrict the access to only a specific common name (CN) of a certificate.

I recently had a use case where an application running inside a pod on an external Kubernetes cluster needed to access AWS resources. Something similar to IAM roles for service accounts (IRSA) would have been ideal, but IRSA cannot work on pods running on external Kubernetes clusters. A very straightforward solution to this problem is to use an IAM user with an access key, but this is not ideal and the security department will not be happy having static credentials being used by the application. Let’s see how we to set up IAM Roles Anywhere to be used from the application running inside a Kubernetes pod.

IAM Roles Anywhere Setup inside AWS

We have to do two steps inside AWS to configure IAM Roles Anywhere:

1. Establish trust: create a trust anchor.
2. Configure IAM roles properly: properly adjust the trust policy and permissions.

In order to establish trust, we can set up a Certificate Authority (CA) using AWS Certificate Manager Private Certificate Authority or use an existing company CA to create a trust anchor. We had an existing CA, so we will be using it in the following steps.

First step: create a trust anchor, choose external certificate bundle, and paste the PEM-encoded certificate bundle in the field.

The second step is to configure the IAM role properly. You can attach the necessary permissions to the role directly. The trust policy will dictate the conditions under which the role can be assumed. Here is an example trust policy that can be used for our use case:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "rolesanywhere.amazonaws.com"
      },
      "Action": [
        "sts:AssumeRole",
        "sts:TagSession",
        "sts:SetSourceIdentity"
      ],
      "Condition": {
        "StringEquals": {
          "aws:PrincipalTag/x509Subject/CN": "myAwesomePodApp",
          "aws:PrincipalTag/x509Issuer/O": "My Awesome Org",
          "aws:PrincipalTag/x509SAN/DNS": "myAwesomePodApp.com",
          "aws:PrincipalTag/x509Subject/O": "My Awesome Org",
          "aws:PrincipalTag/x509Subject/ST": "AwesomeState",
          "aws:PrincipalTag/x509Subject/C": "AwesomeCountry",
          "aws:PrincipalTag/x509Issuer/ST": "AwesomeState",
          "aws:PrincipalTag/x509Issuer/C": "AwesomeCountry",
          "aws:PrincipalTag/x509Issuer/CN": "awesome-internal-ca"
        },
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:rolesanywhere:eu-central-1:123456789012:trust-anchor/c3281693-76c9-48ea-8162-8d48d22c203f"
        }
      }
    }
  ]
}

Notice that the Condition part can list all the details of the certificate that the pod is using such as the common name (CN), the organization (O), the DNS name (DNS), etc. You can decide which parts of the certificate are important for your use case but as a minimum include the CN and DNS details.

After setting up the trust anchor and the IAM role properly, the application holding a valid certificate signed by the CA can assume the role to access AWS resources if the certificate matches the conditions in the trust policy. For example, if the certificate has the common name myAwesomePodApp, and the DNS name myAwesomePodApp.com, is signed by the CA awesome-internal-ca, and all the other specified conitions inside the trust policy, the application can assume the role.

IAM Roles Anywhere Setup on Kubernetes Pods

To use IAM Roles Anywhere and get temporary credentials, we need to install the official credential helper tool. The helper tool manages the process of calling the endpoint to get session credentials and automatically refreshing them before they expire by using the certificate and associated private key. The helper tool has three commands that are interesting for us:

1
2
3

credential-process    Retrieve AWS credentials in the appropriate format for external credential processes
serve                 Serve AWS credentials through a local endpoint
update                Updates a profile in the AWS credentials file with new AWS credentials

The first command credential-process would retrieve the credentials by providing the certificate and private key. The trust anchor arn along the profile arn can be fetched by clicking on the respective details in the IAM Roles Anywhere console. For example, the below command would retrieve the temporary credentials when run from the terminal:

1
2
3
4
5
6

./aws_signing_helper credential-process \
  --certificate ./myAwesomePodApp.crt \
  --private-key ./myAwesomePodApp.key \
  --trust-anchor-arn arn:aws:rolesanywhere:eu-central-1:123456789012:trust-anchor/c3281693-76c9-48ea-8162-8d48d22c203f \
  --profile-arn arn:aws:rolesanywhere:eu-central-1:123456789012:profile/0b3ced02-1f68-4b63-892d-f9ab0117cbfd \
  --role-arn arn:aws:iam::123456789012:role/myAwesomePodAppIAMRole

The update command would obtain the temporary credentials and write them directly to the AWS credentials file. The serve command is a very interesting option for us because it allows us to run a local endpoint that vends temporary security credentials from IAM Roles Anywhere acting like a local token vending machine endpoint. This endpoint can be used by applications running inside the Kubernetes cluster or other VMs to obtain temporary credentials. Our legacy application can benefit from the serve command by consuming the temporary credentials using that endpoint.

Here is an example of the serve command with identical parameters to the command above:

1
2
3
4
5
6

./aws_signing_helper serve \
  --certificate ./myAwesomePodApp.crt \
  --private-key ./myAwesomePodApp.key \
  --trust-anchor-arn arn:aws:rolesanywhere:eu-central-1:123456789012:trust-anchor/c3281693-76c9-48ea-8162-8d48d22c203f \
  --profile-arn arn:aws:rolesanywhere:eu-central-1:123456789012:profile/0b3ced02-1f68-4b63-892d-f9ab0117cbfd \
  --role-arn arn:aws:iam::123456789012:role/myAwesomePodAppIAMRole

The serve command starts a local server that listens on 127.0.0.1 at the default port 9911. We can then use the local endpoint as follows with the aws cli:

1

export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911

After that, you can run the aws cli commands as usual. This is an example if the aws cli is not available and the aws sdk cannot be used by the application for some reason:

1
2
3
4

export AWS_EC2_METADATA_SERVICE_ENDPOINT=http://127.0.0.1:9911
export IAM_ROLE=myAwesomePodAppIAMRole
TOKEN=$(curl -X PUT "$AWS_EC2_METADATA_SERVICE_ENDPOINT/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
curl -H "X-aws-ec2-metadata-token: $TOKEN" $AWS_EC2_METADATA_SERVICE_ENDPOINT/latest/meta-data/iam/security-credentials/$IAM_ROLE

Notice that it implements the same URIs and request headers as IMDSv2. The command will also automatically refresh the credentials five minutes before the previous ones expire.

Architecture Diagram

Here is the architecture diagram of the entire solution:

We have two pods running in the external Kubernetes cluster outside AWS:

• The application pod.
  • It uses the temporary credentials that are fetched by the sidecar container.
• The AWS Signing Helper Pod running as a sidecar container.
  • It uses the certificate and private key mounted as a secret.
  • It uses the parameters needed to call the credential helper tool mounted as a config map.

We discuss the Kubernetes manifest files below.

AWS Signing Helper Pod

The sidecar container for the credential helper tool is a simple container image that uses the official AWS CLI image as the base image:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

FROM public.ecr.aws/aws-cli/aws-cli:2.11.18

ARG IAMRA_VERSION="1.7.3"
ENV IAMRA_VERSION=$IAMRA_VERSION

WORKDIR /usr/local/bin

RUN curl -O "https://rolesanywhere.amazonaws.com/releases/${IAMRA_VERSION}/X86_64/Linux/aws_signing_helper"

RUN chmod a+x aws_signing_helper

USER 1000

For convenience, we will store the built image in ECR and call it iamra_aws_signing_helper.

Certificate Secret Manifest File

Here is the manifest file that stores the certificate and private key:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

---
apiVersion: v1
kind: Secret
metadata:
  name: iamra-myapp-cert
  namespace: myapp
type: kubernetes.io/tls
data:
  tls.crt: |
    LS0tLS1...
  tls.key: |
    LS0tLS1...

Config Map Manifest File

Here is the manifest file that stores the parameters needed to call the credential helper tool:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: iamra-myapp-conf
  namespace: myapp
data:
  IAMRA_ROLE_ARN: "arn:aws:iam::123456789012:role/myAwesomePodAppIAMRole"
  IAMRA_TRUST_ANCHOR_ARN: "arn:aws:rolesanywhere:eu-central-1:123456789012:trust-anchor/c3281693-76c9-48ea-8162-8d48d22c203f"
  IAMRA_PROFILE_ARN: "arn:aws:rolesanywhere:eu-central-1:123456789012:profile/0b3ced02-1f68-4b63-892d-f9ab0117cbfd"

Application Pod Manifest File

Here is the manifest file that runs the application pod and the sidecar container:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

---
apiVersion: v1
kind: Pod
metadata:
  name: iamra-myapp-pod
  namespace: myapp
spec:
  containers:
    - name: myAwesomeApp
      image: 123456789012.dkr.ecr.eu-central-1.amazonaws.com/my_awesome_legacy_app:2.1.34
      command: [ 'sh', '-c' ]
      args:
        - aws sts get-caller-identity;
          aws s3 ls; # access whatever you want here
          ./my_app do_something_with_s3_data; # process the data
          aws s3 cp app_data s3://my-bucket/app_data; # upload to S3
      imagePullPolicy: Always
      env:
        - name: AWS_EC2_METADATA_SERVICE_ENDPOINT
          value: "http://127.0.0.1:9911/"
        - name: AWS_REGION
          value: "eu-central-1"
    - name: iamra-aws-signing-helper
      image: 123456789012.dkr.ecr.eu-central-1.amazonaws.com/iamra_aws_signing_helper:1.7.3
      command: [ "aws_signing_helper" ]
      args:
        - "serve"
        - "--certificate"
        - "/mnt/credentials/tls.crt"
        - "--private-key"
        - "/mnt/credentials/tls.key"
        - "--role-arn"
        - "$(ROLE_ARN)"
        - "--trust-anchor-arn"
        - "$(TA_ARN)"
        - "--profile-arn"
        - "$(PROFILE_ARN)"
      env:
        - name: ROLE_ARN
          valueFrom:
            configMapKeyRef:
              key: IAMRA_ROLE_ARN
              name: iamra-myapp-conf
        - name: TA_ARN
          valueFrom:
            configMapKeyRef:
              key: IAMRA_TRUST_ANCHOR_ARN
              name: iamra-myapp-conf
        - name: PROFILE_ARN
          valueFrom:
            configMapKeyRef:
              key: IAMRA_PROFILE_ARN
              name: iamra-myapp-conf
      volumeMounts:
        - mountPath: /mnt/credentials
          name: credentials
      resources:
        requests:
          memory: "256Mi"
          cpu: "512m"
        limits:
          memory: "512Mi"
          cpu: "1024m"
      imagePullPolicy: Always
  volumes:
    - name: credentials
      secret:
        secretName: iamra-myapp-cert

The iamra-aws-signing-helper container runs the credential helper tool with the parameters from the config map. The application container myAwesomeApp sets the environment variable AWS_EC2_METADATA_SERVICE_ENDPOINT to the local endpoint and uses the aws cli to access S3 resources. The legacy application now can access AWS resources without having to use static credentials. The credential helper tool will automatically refresh the credentials five minutes before expiration time.

Conclusion

In this post, we configured IAM Roles Anywhere and set up the credential helper tool as a sidecar container in a Kubernetes pod to allow a legacy application to access AWS resources without using static credentials from an IAM user. We used the application certificate signed by our own private CA to fetch the temporary credentials from IAM Roles Anywhere. The application could then consume these credentials and access the needed AWS resources.

References:

• aws-samples/aws-iam-ra-for-kubernetes
• Sidecar Containers
• Introducing fine-grained IAM roles for service accounts
• Diving into IAM Roles for Service Accounts