Automatic Management of AWS ECR Credentials in a Kubernetes Cluster

Marco Ferraioli — Wed, 21 Jun 2023 11:55:00 +0000

In the course of my work with AWS ECR (Elastic Container Registry), I ran into a problem: The repository access key expires every six hours. Working with a non-AWS Kubernetes test cluster, I had to constantly update these credentials manually, a repetitive and tedious process.

From this experience came the idea to create a tool that automated this process: k8s-aws-ecr-secret-updater. This tool is a Kubernetes cronjob, designed to automatically update the AWS ECR repository access credentials.

Cronjob Configuration

The YAML code to create the cronjob consists of several parts, which I will now analyze piece by piece:

A Role that has permission to get, create, and delete secrets and get and update ServiceAccounts.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: k8sawsecrsecretupdater
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "create", "delete"]
- apiGroups: [""]
  resources: ["serviceaccounts"]
  verbs: ["get", "patch"]

A key component of this configuration is the k8sawsecrsecretupdater role. This role is fundamental for authorization within the Kubernetes namespace, allowing the cronjob to perform specific operations on certain resources.

In particular, the k8sawsecrsecretupdater role has the following permissions:

It has permissions to get (get), create (create), and delete (delete) Secrets. This is crucial because the cronjob needs to be able to create and delete AWS ECR credentials, which are stored as secrets in Kubernetes.
It has permissions to get (get) and update (patch) ServiceAccounts. The cronjob needs to be able to manage service accounts in order to associate the AWS ECR credentials with the service that runs the cronjob.

Creating a specific role for these operations ensures that the cronjob has exactly the permissions it needs to do its job, without granting it access to unnecessary resources. This approach is in line with the principle of least privilege, a common security practice that limits access to resources only to what is strictly necessary to perform a specific task. This helps to minimize the potential impact of a possible attack.

ServiceAccount to be used by the job and cronjob.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8sawsecrsecretupdater

RoleBinding to associate the Role with the ServiceAccount.

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: k8sawsecrsecretupdater
subjects:
  - kind: ServiceAccount
    name: k8sawsecrsecretupdater
roleRef:
  kind: Role
  name: k8sawsecrsecretupdater
  apiGroup: rbac.authorization.k8s.io

Job that creates the secret.

apiVersion: batch/v1
kind: Job
metadata:
  name: k8sawsecrsecretupdater
spec:
  backoffLimit: 4
  template:
    spec:
      serviceAccountName: k8sawsecrsecretupdater
      restartPolicy: Never
      containers:
      - name: k8sawsecrsecretupdater
        image: ghcr.io/paranoiasystem/k8s-aws-ecr-secret-updater:latest
        imagePullPolicy: Always
        env:
        - name: AWS_ACCOUNT
          value: 'YourAwsAccountID'
        - name: AWS_ACCESS_KEY_ID
          value: YourAccessKeyID
        - name: AWS_SECRET_ACCESS_KEY
          value: YourSecretAccessKey
        - name: AWS_REGION
          value: YourRegion

CronJob that runs the Job every 6 hours.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: k8sawsecrsecretupdater
spec:
  schedule: "0 */6 * * *"
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: k8sawsecrsecretupdater
          restartPolicy: Never
          containers:
          - name: k8sawsecrsecretupdater
            image: ghcr.io/paranoiasystem/k8s-aws-ecr-secret-updater:latest
            imagePullPolicy: Always
            env:
            - name: AWS_ACCOUNT
              value: 'YourAwsAccountID'
            - name: AWS_ACCESS_KEY_ID
              value: YourAccess
            - name: AWS_SECRET_ACCESS_KEY
              value: YourSecretAccessKey
            - name: AWS_REGION
              value: YourRegion

Creating the Docker Image

The cronjob uses a specific Docker image to perform its task. This Docker image is built from the following Dockerfile:

In the Dockerfile, starting from an Alpine base image, the necessary tools are installed, including git, bash, curl, openssh, python3, py3-pip, py-cryptography, wget, curl, jq. Also, kubectl and AWSCLI are installed for interaction with Kubernetes and AWS, respectively.

Subsequently, the working directory is set to /scripts and the contents of the local /scripts directory are copied. Finally, an ENTRYPOINT is defined that starts the entrypoint.sh script when the container is run.

Execution of the Script

When the cronjob is run, it starts the bash script contained in the Docker image. This script checks for the existence of a secret called "regcred". If it exists, it deletes it and creates a new one. If it doesn't exist, it creates it. Below is the script:

#!/bin/bash

create_secret() {
  kubectl create secret docker-registry regcred \
    --docker-server=${AWS_ACCOUNT}.dkr.ecr.${AWS_REGION}.amazonaws.com \
    --docker-username=AWS \
    --docker-password=$(aws ecr get-login-password --region ${AWS_REGION})
}

# Check if the secret exists
if kubectl get secret regcred; then
  # If it exists, delete it
  kubectl delete secret regcred
  # Create the secret again
  create_secret
else
  # If it doesn't exist, create it
  create_secret
fi

Installing k8s-aws-ecr-secret-updater on Kubernetes

To use the k8s-aws-ecr-secret-updater in your Kubernetes environment, you need to follow some simple steps.

Let's start by cloning the project's GitHub repository onto your local system:

git clone https://github.com/paranoiasystem/k8s-aws-ecr-secret-updater

Next, you need to open and edit the install.yaml file present in the repository. In this file, you will need to set the following values:

AWS_ACCOUNT: your AWS account ID.
AWS_ACCESS_KEY_ID: your AWS access key.
AWS_SECRET_ACCESS_KEY: your AWS secret access key.
AWS_REGION: the AWS region where your ECR is located.

Once you have made these changes, save and exit the install.yaml file.

Now, to install the k8s-aws-ecr-secret-updater in your Kubernetes cluster, you need to execute the following command:

kubectl apply -n <destination_namespace> -f install.yaml

Remember to replace destination_namespace with the Kubernetes namespace where you want to install the cronjob.

Conclusions

This tool eliminates the need for manual updating of credentials, saving time and reducing the risk of errors. I hope this article and the tool I created can be of help to anyone dealing with a similar issue in managing AWS ECR credentials in a Kubernetes cluster.

Forem: Marco Ferraioli