Forem: Mark van Holsteijn

How to overcome Docker Hub rate limiting using AWS ECR and AWS CloudFormation

Mark van Holsteijn — Sun, 08 Oct 2023 22:00:00 +0000

In this blog post, you will see how AWS ECR and AWS CloudFormation overcome the rate limiting imposed by Docker Hub and provide full control over your base images.

The popular registry Docker Hub is home to thousands of useful container images, used by many software delivery processes. Unfortunately, the registry enforces a rate limit for anonymous and free-tier users. Whenever you try to pull an image from an AWS CodeBuild project, you immediately run it this problem. AWS offers many Docker Hub images directly from their public AWS ECR registry https://public.ecr.aws, but not all of them. So, when you want to use a public image not on offer, you are stuck.

Overcome Docker Hub rate limiting

Our Custom CloudFormation Container Image Provider offers an effective workaround by allowing you to clone public images into a private Amazon Elastic Container Registry repository. By leveraging the custom provider you use CloudFormation to avoid the rate limit imposed by Docker Hub. Once the image is copied, you can pull the image as often as you want from your own ECR repository. This ensures a smooth and uninterrupted development process.

Full control over updating base images

Another advantage of using the Custom CloudFormation Container Image Provider is that you gain complete control over the base images. You can enable container image scanning and see which vulnerabilities live inside the public image. By using a CloudFormation template you specify the exact image version you want.

The container reference update utility – cru can be used to updates image references in the CloudFormation template and trigger the provisioning of the latest version to your ECR repository.

This effectively gives you a well defined process for provisioning container images.

Example usage

To demonstrate the usage of the Custom CloudFormation Container Image Provider, let’s consider the following CloudFormation template:

Resources:
  Repository:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: python

  Python37:
    Type: 'Custom::ContainerImage'
    Properties:
      ImageReference: docker.io/library/python:3.7
      RepositoryArn: !GetAtt Repository.Arn
      ServiceToken: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:cfn-container-image-provider'

In this example, we clone the current repository from the public image ‘python:3.7’ into our ‘python’ repository in ECR. The ‘Repository’ resource creates the ECR repository, and the ‘Python37’ resource uses the custom resource ‘Custom::ContainerImage’ to clone the image.

Updating the image reference

To pin the image to a specific version, you can use the container reference update utility – cru as follows:

$ cru update \
        --resolve-digest --all \
        --matching-tag \
        demo.yaml

023/10/07 16:20:56 INFO: 1 image references found
2023/10/07 16:20:57 resolving repository docker.io/library/python Tag 3.7 to Digest sha256:eedf63967cdb57d8214db38ce21f105003ed4e4d0358f02bedc057341bcf92a0
2023/10/07 16:20:57 INFO: updated a total of 1 files
2023/10/07 16:20:57 INFO: no commit message, skipping commit and push

Now the container image reference will have the associated digest of the image, so you know exactly which image is used.

Python37:
    Type: 'Custom::ContainerImage'
    Properties:
      ImageReference: 'docker.io/library/python:3.7@sha256:eedf63967cdb57d8214db38ce21f105003ed4e4d0358f02bedc057341bcf92a0'

Installing the provider

To install this custom resource provider, type:

read -p 'VPC id:' VPC_ID
read -p 'private subnet ids (comma separated):' PRIVATE_SUBNET_IDS
read -p 'security group ids (comma separated):' SECURITY_GROUP_IDS

aws cloudformation create-stack \
       --capabilities CAPABILITY_IAM \
       --stack-name cfn-container-image-provider \
       --template-url s3://binxio-public-eu-central-1/lambdas/cfn-container-image-provider-0.2.4.yaml \
       --parameter-overrides \
          ParameterKey=AppVPC,ParameterValue=$VPC_ID \
          ParameterKey=Subnets,ParameterValue=$PRIVATE_SUBNET_IDS \
          ParameterKey=SecurityGroupIds,ParameterValue=$SECURITY_GROUP_IDS

aws cloudformation wait stack-create-complete \
       --stack-name cfn-container-image-provider

or use launch stack.

The provider is installed on the private subnets in your VPC, to ensure that your NAT gateway IP addresses are used to pull images from docker hub.

Conclusion

The Custom CloudFormation Container Image Provider addresses two important challenges that developers and organisations face when working with container images. By cloning public images into your ECR repository, you can overcome the rate limit imposed by Docker Hub, and ensure uninterrupted access to the images you need. Additionally, you gain full control over which images are used in your organisation.

Photo by Zé Maria on Unsplash

The post How to overcome Docker Hub rate limiting using AWS ECR and AWS CloudFormation appeared first on Xebia.

Seamlessly connect Gitlab CI/CD with AWS: Gitlab AWS credential helper

Mark van Holsteijn — Sun, 01 Oct 2023 22:08:00 +0000

In this blog we will show you how our Gitlab AWS credential helper will make it very easy to connect to an AWS account using the gitlab pipeline id token as your identity.

Introduction

Since June 2022, Gitlab provides JWT identity tokens for each project pipeline, which can be exchanged for AWS access credentials using the assume-role-with-web-identity API call. This is great, because it allows you to configure the AWS permissions for each individual pipeline. However the problem is that exchanging the token for credentials requires quite a bit of shell programming. Something along the following lines:

ROLE_ARN=$1
WEB_IDENTITY_TOKEN=$2
ROLE_SESSION_NAME=gitlab-${CI_PROJECT_PATH_SLUG}-${CI_PIPELINE_ID}

CREDENTIALS=($(aws sts assume-role-with-web-identity \
        --role-arn ${ROLE_ARN} \
        --role-session-name ${ROLE_SESSION_NAME} \
        --web-identity-token "${WEB_IDENTITY_TOKEN}" \
        --duration-seconds 3600 \
        --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
        --output text))

aws configure set aws_access_key_id "${CREDENTIALS[0]}"
aws configure set aws_secret_access_key "${CREDENTIALS[1]}"
aws configure set aws_session_token "${CREDENTIALS[2]}"

As you can see that is quite a complex shell script. This will not look pretty in your .gitlab-ci.yml, and you will would probably add this as a separate script to your project. For a single project this is ok-ish, but to apply that to all your projects, is not going to scale well: As this script is not suitable for all situations, you probably end up with some variants as well.

Gitlab AWS credential helper to the rescue

This is where the gitlab aws credential helper comes in! The utility is flexible and very easy to use in all sorts of situations. It is simple to configure, as it only needs the AWS account number and Gitlab id token. The Gitlab project path slug and pipeline id will be used to determine the IAM role- and session name. The general configuration of the CI/CD pipeline consists of the following five steps:

Add the Gitlab ID token
Set the AWS variables
Configure the credential helper
Create the IAM role for the pipeline
Install the credential helper

Let’s us take you through these five steps in the following paragraphs.

Add Gitlab ID token

To obtain an Gitlab ID token in your pipeline jobs, add the following definition to your pipeline definition:

default:
  id_tokens:
    GITLAB_AWS_IDENTITY_TOKEN:
      aud: https://gitlab.com

This will cause each job to have an environment variable name GITLAB_AWS_IDENTITY_TOKEN with the JWT identity token signed by Gitlab as a value. The credential helper expects the token in this variable.

Set AWS variables

To know which AWS account to connect to, add the environment variable GITLAB_AWS_ACCOUNT_ID with your AWS account id:

variables:
  GITLAB_AWS_ACCOUNT_ID: '1234567898012'

If you have many gitlab projects, you can also set this variable as a CI/CD variable on the group level. We recommend to add the following variables as well:

variables:
  - AWS_CONFIG_FILE: ${CI_PROJECT_DIR}/.aws/config
  - AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}/.aws/credentials
  - AWS_PROFILE: default

These will make sure the credential configurations are kept in scope of the pipeline build and that the obtained credentials are actually used.

Using the credential helper

Now you are ready to use the credential helper, in one of three different ways:

as credential process
stored as shared credentials
as environment variables

Let’s look at each of these options.

As credential process

The most elegant use, is to configure the credential helper as AWS credential process, like shown below:

before_script:
  - >
    aws config --profile default 
    set credential_process 
    "gitlab-aws-credential-helper process"

script:
  - aws --profile default sts get-caller-identity

The AWS SDK will call the credential helper whenever it requires to access to AWS!

Stored as shared credentials

Alternatively, the helper can store the AWS credentials directly in the AWS shared credentials file. You can do this as follows:

before_script:
  - >
    gitlab-aws-credential-helper aws-profile 

script:
  - aws --profile default sts get-caller-identity

This will store the credentials in shared credentials file, normally ~/.aws/credentials.

Get as environment variables

Finally, you can use the helper to export the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY

and AWS_SESSION_TOKEN, as shown below:

script:
  - eval $(gitlab-aws-credential-helper env --export)
  - aws sts get-caller-identity

Alternatively, you can directly execute the command:

script:
  - gitlab-aws-credential-helper env -- aws sts get-caller-identity

Create the pipeline IAM role

Once the pipeline is prepared, you need to add the IAM role for the CI/CD pipeline. The name of the role should be equal to the name of the gitlab project path slug as defined by the variable CI_PROJECT_PATH_SLUG, prefixed with gitlab-. The following CloudFormation template shows an example IAM role definition:

AWSTemplateFormatVersion: '2010-09-09'
Description: Role for gitlab pipeline with limited AWS access
Parameters:
  RepositoryPath:
    Description: the gitlab repository path
    Type: String
  RepositoryPathSlug:
    Description: the gitlab repository path slug
    Type: String
Resources:
  GitlabPipeline:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub 'gitlab-${RepositoryPathSlug}'
      MaxSessionDuration: 3600
      AssumeRolePolicyDocument:
        Statement:
          - Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Sub 'arn:aws:iam::${AWS::AccountId}:oidc-provider/gitlab.com'
            Effect: Allow
            Condition:
              ForAnyValue:StringLike:
                "gitlab.com:sub":
                  - !Sub "project_path:${RepositoryPath}:ref_type:branch:ref:*"
      Policies:
        - PolicyName: MetaInformationAccess
          PolicyDocument:
            Statement:
              - Effect: Allow
                Action:
                  - sts:GetCallerIdentity
                Resource:
                  - '*'

The template assumes you have already created the OIDC provider for gitlab.com in your account. If not, checkout this blog.

To create the role you need, type:

CI_PROJECT_PATH=mvanholsteijn/aws-credential-helper-demo
CI_PROJECT_PATH_SLUG=mvanholsteijn-aws-credential-helper
aws cloudformation deploy \
  --stack-name gitlab-$CI_PROJECT_PATH_SLUG \
  --template-file gitlab-pipeline-limited-access.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
   --parameter-overrides \
     "RepositoryPath=$CI_PROJECT_PATH" \
     "RepositoryPathSlug=$CI_PROJECT_PATH_SLUG"

This all you need for one pipeline. The following two paragraphs will explain how to install the credential helper and show you how to override defaults.

Install the credential helper

Of course, your job will need to have the gitlab-aws-credential-helper installed. We recommend to add it to the image that runs your job using a multi-stage docker build file, as show below:

FROM ghcr.io/binxio/gitlab-aws-credential-helper:0.3.2 as gitlab-aws-credential-helper

FROM <yourimage>

COPY --from=gitlab-aws-credential-helper /usr/local/bin/gitlab-aws-credential-helper /usr/local/bin

If you do not manage your own runner image, you can add the following command to the before_script section.

NAME=gitlab-aws-credential-helper
RELEASE=0.3.2
URL=https://github.com/binxio/$NAME/releases/download/${RELEASE}/${NAME}_${RELEASE}_linux_amd64.tar.gz
CHECKSUM=b8c84f1ca5622f4b0f529eca74e7689ea20418eab31a96666366ae1e1531b41f
ARCHIVE="$(basename $URL)"

cd /tmp && \
    curl -L -O -sS $URL && \
    (echo "$CHECKSUM $ARCHIVE" | sha256sum -c -) && \
    tar -C /usr/local/bin/ -xzf $ARCHIVE && \
    rm -f $ARCHIVE

Overriding defaults

The credential helper uses sensible defaults to make it as easy as possible. Of course, you can override all options via environment variables or via the command line. The following example shows how to create different profiles for a named role in different accounts:

before_script:
  ->
    gitlab-aws-credential-helper
    aws-profile --name development
    --aws-account 111111111111 
    --role Deployer
  ->
    gitlab-aws-credential-helper
    aws-profile --name test
    --aws-account 222222222222 
    --role Deployer
  ->
    gitlab-aws-credential-helper
    aws-profile --name production
    --aws-account 333333333333 
    --role Deployer

Use the --help flag to get an overview of all possible options for each of the command usages.

Conclusion

With the Gitlab AWS credential helper, it becomes really easy to provide each Gitlab CI/CD pipeline with the least amount of privileges required to perform their task on AWS. Just configure the AWS account number and add the credential helper to the runner and create an IAM role for each gitlab project.

The post Seamlessly connect Gitlab CI/CD with AWS: Gitlab AWS credential helper appeared first on Xebia.

How to create and deploy a golang AWS CloudFormation custom provider in less than 5 minutes

Mark van Holsteijn — Mon, 31 Jul 2023 22:16:00 +0000

In this blog I will show you how to create and deploy a Golang AWS CloudFormation custom provider in less than 5 minutes using a copier template.

Creating a custom resource in CloudFormation is really simple. You just implement a create, update and delete method in a Lambda and you are done. But that is the easy part: you still have to create zip files, unit tests, documentation, demo’s, CI/CD deployment pipelines and more. This copier template has it all!

it creates the source code for a Golang custom resource provider
it provides all the build commands you need to build, test and deploy your provider
it deploys lambdas to buckets in all AWS regions in the world
it provides a ready-to-deploy CI/CD pipeline on AWS Codebuild
it supports semantic versioning using git-release-tag

getting started!

Let’s say you want to create a custom resource for a ECR Container Image, so that you an clone public images into an ECR repository. Although there is a Python library to implement this, it is unfortunately no longer supported. Fortunately, there is a Golang library go-containerregistry to do the trick. To skaffold a project to implement a custom provider based on Golang, type:

$ pip install 'copier>=8.0.0'
$ copier copy --trust \
   https://github.com/binxio/cloudformation-custom-provider-golang-template \
   /tmp/cfn-container-image-provider

🎤 the name of your custom resource type?
   ContainerImage
🎤 The name of your resource provider?
   cfn-container-image-provider
🎤 a short description for the custom provider?
   manages container images
🎤 golang version to use
   1.20
🎤 Your full name?
   Mark van Holsteijn
🎤 Your email address?
   mark.vanholsteijn@xebia.com
🎤 the go module name
   github.com/binxio/cfn-container-image-provider
🎤 the URL to git source repository?
   https://github.com/binxio/cfn-container-image-provider.git
🎤 the AWS region name
   eu-central-1
🎤 prefix for the S3 bucket name to store the lambda zipfiles?
   binxio-public
🎤 Access to lambda zip files?
   public

Copying from template version 0.1.0
 ...
 > Running task 1 of 1: [! -f go.sum] && (go mod download || echo "WARNING: failed to run go mod">&2); [! -d .git] && ( git init && git add . && git commit -m 'initial import' && git tag 0.0.0) || exit 0
Initialized empty Git repository in ...
[main (root-commit) c97b9e2] initial import
 15 files changed, 529 insertions(+)

source code directory

The copier generates the following source code directory:

├── Dockerfile.lambda # creates the zip file
├── Makefile.mk # generic build steps for the provider
├── Makefile # customization of build steps
├── cloudformation
│ ├── cfn-custom-image-provider.yaml # to deploy the provider
│ ├── cicd-pipeline.yaml # to deploy the Codebuild CI/CD pipeline
│ └── demo.yaml # to deploy the demo
├── doc
│ └── ContainerImage.md # start documentation of resource
├── go.mod
├── go.sum
├── main.go # main entrypoint
└── pkg
    └── resources
        └── container_image
            └── handler.go # sample code

That is all that is needed to create a project with a working custom provider for the resource ContainerImage. When you change to the directory and type:

$ make deploy-provider
$ make deploy-demo

Your provider will be up-and-running in less than 5 minutes! Now it is up to you refactor the implementation to provide the required functionality.

Available build commands

to help you in the development process, the following build commands are available:

$ make help

build - build the lambda zip file
fmt - formats the source code

test - run unit tests
test-templates - validate CloudFormation templates

deploy - AWS lambda zipfile to bucket
deploy-all-regions - AWS lambda zipfiles to all regional buckets
undeploy-all-regions - deletes AWS lambda zipfile of this release from all
buckets in all regions

deploy-provider - deploys the custom provider
delete-provider - deletes the custom provider

deploy-pipeline - deploys the CI/CD deployment pipeline
delete-pipeline - deletes the CI/CD deployment pipeline

deploy-demo - deploys the demo stack
delete-demo - deletes the demo stack

tag-patch-release - create a tag for a new patch release
tag-minor-release - create a tag for a new minor release
tag-major-release - create a tag for new major release

show-version - shows the current version of the workspace
help - Show this help.

Deploy the zip file to the bucket

To copy the zip file with the source code of the AWS Lambda of the custom resource provider, the buckets must already exist. If they do not, type:

$ BUCKET=<bucket-prefix>-<bucket-region>
$ aws s3 mb s3://$BUCKET
$ aws s3api put-bucket-ownership-controls \
    --bucket $BUCKET --ownership-controls \
    'Rules=[{ObjectOwnership=BucketOwnerPreferred}]'

The build system expects the bucket name to consists of the prefix and the region name. This allows the provider to be made available for use in all regions.

Deploy the custom resource provider into the account

To configure the run-time parameters and permissions of the your provider, change the CloudFormation template in the directory ./cloudformation. Once that is done, type:

$ make deploy-provider

Deploy the custom resource demo

To deploy a CloudFormation stack with an example use of the custom resource, type:

$ make deploy-demo

Version your custom resource provider

Semantic versioning of the provider is implemented using the utility git-release-tag. If you have not installed git-release-tag, type:

$ pip install git-release-tag

To version your custom resource provider, you can use the following commands:

make tag-patch-release - create a tag for a new patch release
make tag-minor-release - create a tag for a new minor release
make tag-major-release - create a tag for new major release

This will:

run the pre-tag command in the file ./release, to update all files with references to the semantic version
commit all outstanding changes in the workspace
tag the commit with the new version

To show the current version of the workspace, type:

make show-version

Deploy provider to all regions

To deploy the current version of your provider to all regions, type:

make deploy-all-regions

This assumes you have buckets in all regions with the defined prefix.

Deploy CI/CD pipeline

To deploy the CI/CD pipeline based on AWS Codebuild, make sure that the AWS account can access the source repository. If that is the case, type:

make deploy-pipeline

Now, every time you tag a new release, it will automatically be deployed to all regions.

Conclusion

This copier template provides everything you need to quickly build, deploy and maintain a new Golang custom AWS CloudFormation Provider! If you have any questions, problems or feedback feel free to contact me or add issues at https://github.com/binxio/cloudformation-custom-provider-golang-template.

If you want to create your resource provider in Python, we also have a solution for that!

Image by Michal Jarmoluk from Pixabay

The post How to create and deploy a golang AWS CloudFormation custom provider in less than 5 minutes appeared first on Xebia.

How to create and deploy a Slackbot on Google Cloud Platform in less than 5 minutes

Mark van Holsteijn — Sun, 19 Mar 2023 09:39:00 +0000

In this blog I will show you how to create and deploy a Slackbot on Google Cloud Platform in less than 5 minutes using copier template. To write a bot for Slack is not too difficult. With a few lines of code, you can create a program which will respond to slack commands and mentions. But that is the easy part: you also need to configure the application in Slack, deploy it so that is available 7×24, write unit tests and create CI/CD deployment pipelines and more. This copier template has it all!

it creates the source code for your slackbot.
it provides the Terraform templates for the infrastructure, the application and the CI/CD pipeline.
it provides a CI/CD pipeline using Google CloudBuild to build and deploy the application and infrastructure changes.

This means you can start interacting with your slackbot and iteratively improve the implementation until you have the perfect bot.

deploy a slackbot to Google Cloud Platform

To create and deploy your slackbot, you need to:

generate the source directory
create the application in Slack
install the application in the workspace
obtain the application token
obtain the signing secret
obtain the bot user OAuth token
deploy the slackbot
deploy the tokens and secrets
run the CI/CD pipeline

The following sections provide a detailed explanation of each step.

generate the source directory

Let’s say you want to create a slackbot which generates images using Dall-e. To generate the source directory, type:

$ pip install copier
$ copier https://github.com/binxio/slackbot-on-google-cloud-platform-template.git /tmp/dali


> the human readable name of your slackbot
   Salvador Dali
> a short description of the Slackbot
   generates images on request
> the name of the package
   salvador_dali_slackbot
> the slackbot command prefix
   /dali
> Your full name?
   Mark van Holsteijn
> Your email address?
   mark.vanholsteijn@xebia.com
> the google project to deploy to
   speeltuin-mvanholsteijn
> primary region to deploy the slackbot to
   europe-west4
> the replica region for slackbot artifacts
   europe-west1
> the artifact repository location to deploy the image to
   europe
> name of the terraform state bucket
   speeltuin-mvanholsteijn-terraform-state

Now the source code is ready in /tmp/dali. The contents of that directory looks like this:

.
├── src # bootstrap implementation
│   └── salvador_dali_slackbot
│   ├── __init__.py
│   ├── __main__.py
│   ├── bot.py
│   ├── google_secrets.py
│   └── manifest.yaml
├── tests # skaffold tests for bot functions
│   ├── test_command_help.py
│   └── test_mention_help.py
├── terraform # deployment definitions
│   ├── pipeline.tf
│   ├── slackbot.tf
│   ├── user-data.yaml
│   ├── variables.tf
│   └── versions.tf
│   ├── providers.tf
├── cloudbuild # CI/CD pipeline definitions
│   ├── build.yaml
│   └── deploy.yaml
├── secrets 
├── Dockerfile
├── Makefile
├── Makefile.mk
├── Pipfile
├── pyproject.toml
├── setup.cfg
├── tox.ini

As you can see everything you need to build, maintain and deploy a slackbot is there! Next, you need to create the application in Slack.

Create the application in Slack

To create the slackbot in Slack, you can use the application manifest manifest.yaml found in the generated source directory. Copy the contents and goto https://api.slack.com/apps and click on “create app from manifest”.

your next step is to install the application in the slack workspace.

Install the application in the workspace

After you created the application, you can install it in your workspace. It is easy: just click the button and follow the flow.

Once the application is installed, you need to obtain the application token, signing secret and bot token.

Obtain the application token

After you install the application, scroll down the screen and generate an application token with the scope connections:write. Copy the generated app token into the file secrets/app-token. This token will put into a Google secret manager secret later. First you need to get the signing secret and bot token.

Obtain the application signing secret

To obtain the application signing secret, Navigate to Application Basic information, and copy the signing secret into the file secrets/signing-secret. This secret will put into a Google secret manager secret later. The bot token is the last secret we need to get, before we can deploy the bot.

Obtain the bot user OAuth token

To obtain the bot user OAuth token, navigate to OAuth and permissions and copy the token into the file secrets/bot-token.

Now everything is done to deploy the Slackbot using Terraform

deploy the slackbot

Before we can deploy the slackbot to Google Cloud Platform, you need to enable the used services and create the Terraform storage bucket. Type:

$ make enable-services 
$ make state-bucket

Now you are ready to let Terraform do it’s magic. Type:

$ cd terraform
$ terraform init
$ terraform apply

This creates everything you need, but the bot will not yet run. The tokens and secrets needs to be made available as Google secrets.

deploy the tokens and secrets

To deploy the tokens and secrets of the slackbot into the respective Google secrets, type:

$ make configure-secrets

run the CI/CD pipeline

Now deploy your changes to the git repository. This will trigger a build of the container image and a deployment of the newly created image using Google CloudBuild.

$ git push

Check the Google CloudBuild logs Once the build and deploy complete your bot is ready to run! You can now chat with your bot and start developing the functionality.

Conclusion

This copier template provides everything you need to quickly build, deploy and maintain a Slackbot on Google Cloud Platform. If you have any questions, problems or feedback feel free to contact me. You can also directly add issues on Github

Photo by Stephen Phillips – Hostreviews.co.uk on Unsplash

The post How to create and deploy a Slackbot on Google Cloud Platform in less than 5 minutes appeared first on Xebia.

How to create and deploy an AWS CloudFormation custom provider in less than 5 minutes

Mark van Holsteijn — Wed, 08 Mar 2023 23:08:00 +0000

In this blog I will show you how to create and deploy an AWS CloudFormation custom provider in less than 5 minutes using a Python copier template. To create a custom resource in CloudFormation, is really simple. You just implement a create, update and delete method in a Lambda and you are done. But that is the easy part: you still have to create zip files, unit tests, documentation, demo’s, CI/CD deployment pipelines and more. This copier template has it all!

it creates the source code for a custom resource provider
all the build commands you need to build, test and deploy your provider
it provides re-recordable unit testsusing botocore stubber recorder
it deploys lambdas to buckets in all AWS regions in the world
it supports semantic versioning using git-release-tag
it provides a ready-to-deploy CI/CD pipeline on AWS Codebuild

getting started!

Let’s say you want to create a custom resource for a Custom Domain of an AWS AppRunner service, because it does not yet exist. To get started, type:

$ pip install copier
$ copier https://github.com/binxio/cloudformation-custom-provider-template /tmp/cfn-app-runner-custom-domain-provider

🎤 the name of your custom resource type?
   AppRunnerCustomDomain
🎤 The name of your resource provider project?
   cfn-app-runner-custom-domain-provider
🎤 The name of your Python module?
   cfn_app_runner_custom_domain_provider
🎤 a short description for the custom provider?
   manages app runner custom domains
🎤 Python version to use
   3.9
🎤 Your full name?
   Mark van Holsteijn
🎤 Your email address?
   mark.vanholsteijn@xebia.com
🎤 the URL to git source repository?
   https://github.com/binxio/cfn-app-runner-custom-domain-provider
🎤 the AWS profile name
   integration-test
🎤 the AWS region name
   eu-central-1
🎤 prefix for the S3 bucket name to store the lambda zipfiles?
   binxio-public
🎤 Access to lambda zip files?
   public

> Running task 1 of 1: [[! -d .git]] && ( git init && git add . && git commit -m 'initial import' && git tag 0.0.0) || exit 0
Initialized empty Git repository in /tmp/cfn-app-runner-custom-domain-provider/.git/
[main (root-commit) b2ce863] initial import
 21 files changed, 619 insertions(+)
...

source code directory

The copier generates the following source code directory:

.
├── Dockerfile.lambda # creates the zip file
├── Makefile.mk # generic build steps for the provider
├── Makefile # customization of build steps
├── Pipfile # python virtual environment definition
├── cloudformation
│   ├── cfn-app-runner-custom-domain-provider.yaml # to deploy the provider
│   ├── cicd-pipeline.yaml # to deploy the Codebuild CI/CD pipeline
│   └── demo.yaml # to deploy the demo
├── doc
│   └── AppRunnerCustomDomain.md # start documentation of resource
├── pyproject.toml # Python project definition
├── setup.cfg # Python package settings
├── tox.ini # Python test definitions
├── src
│   └── cfn_app_runner_custom_domain_provider
│   ├── __init__.py
│   ├── app_runner_custom_domain.py # custom resource definition source
│   └── logger.py
├── tests
│   └── crud
│   ├── __init__.py
│   ├── base
│   │   └── __init__.py
│   └── test_crud.py # custom resource definition test

That is all that is needed to create a project with a working custom provider for the resource AppRunnerCustomDomain. When you change to the directory and type:

$ pipenv install -d
$ make deploy-provider
$ make demo

Your provider will be up-and-running in less than 5 minutes!

Available build commands

to see a list of all of available build commands, type:

$ make help

build - build the lambda zip file
fmt - formats the source code using black
test - run python unit tests
test-record - run python unit tests, while recording the boto3 calls
test-templates - validate CloudFormation templates

deploy - AWS lambda zipfile to bucket
deploy-all-regions - AWS lambda zipfiles to all regional buckets
undeploy-all-regions - deletes AWS lambda zipfile of this release from all buckets in all regions

deploy-provider - deploys the custom provider
delete-provider - deletes the custom provider

deploy-pipeline - deploys the CI/CD deployment pipeline
delete-pipeline - deletes the CI/CD deployment pipeline

deploy-demo - deploys the demo stack
delete-demo - deletes the demo stack

tag-patch-release - create a tag for a new patch release
tag-minor-release - create a tag for a new minor release
tag-major-release - create a tag for new major release

Run the unit tests

To run the unit tests, type:

$ make test

The unit test will test the scaffold implementation generated by the botocore stubber recorder. To create unit tests for your resource, edit the source code in ./tests/. To implement your custom resource, edit the source code under ./src/.

Re-recordable unit tests

Once you have your custom resource provider, it undoubtedly does some AWS API calls. The botocore stubber recorder library records the actual calls and generate the stub. The unit tests can be run in recording mode, where the test is against a real AWS account. To run your unit tests as integration test against a live system and record the stubs, type:

$ make test-record

To run the unit tests with the newly created stubs, type:

$ make test

The integration tests are run against the AWS profile and region you specified. To add a new test, type:

$ botocore-stubber-recorder --test-name failed_create

Deploy the zip file to the bucket

To copy the zip file with the source code of the AWS Lambda of the custom resource provider, the buckets must already exist. If they do not, type:

$ aws s3 mb s3://<bucket-prefix>-<bucket-region>

The build system expects the bucket name to consists of the prefix and the region name. This allows the provider to be made available for use in all regions.

Deploy the custom resource provider into the account

To configure the run-time parameters and permissions of the your provider, change the CloudFormation template in the directory ./cloudformation. Once that is done, type:

$ make deploy-provider

Deploy the custom resource demo

To deploy a CloudFormation stack with an example use of the custom resource, type:

$ make deploy-demo

Version your custom resource provider

Semantic versioning of the provider is implemented using the utility git-release-tag. To version your custom resource provider, you can use the following commands:

make tag-patch-release - create a tag for a new patch release
make tag-minor-release - create a tag for a new minor release
make tag-major-release - create a tag for new major release

This will: * runs the pre-tag command in the file ./release, to update all files with references to the semantic version * commit all outstanding changes in the workspace * tag the commit with the new version. To show the current version of the workspace, type:

make show-version

Deploy provider to all regions

To deploy the current version of your provider to all regions, type:

make deploy-all-regions

This assumes you have buckets in all regions with the defined prefix.

Deploy CI/CD pipeline

To deploy the CI/CD pipeline based on AWS Codebuild, make sure that the AWS account can access the source repository. If that is the case, type:

make deploy-pipeline

Now, every time you tag a new release, it will automatically be deployed to all regions.

Conclusion

This copier template provides everything you need to quickly build, deploy and maintain a new custom AWS CloudFormation Provider! If you have any questions, problems or feedback feel free to contact me or add issues at https://github.com/binxio/cloudformation-custom-provider-template. If you want to create your resource provider in Golang, we also have a solution for that!

Photo by Saffu on Unsplash

The post How to create and deploy an AWS CloudFormation custom provider in less than 5 minutes appeared first on Xebia.

How to start a RDP session from the command line to a Windows server running on AWS

Mark van Holsteijn — Tue, 25 Oct 2022 18:00:00 +0000

To start a RDP session to a Windows server on AWS is a very labour-intensive task. You have to select the instance on the console, copy the private key to get the password, copy the password, download the RDP file. Then double-click on the RDP file, paste the password in a dialog box, and you are done. But it does not have to be this way. In this blog we will show you it can be as easy as using ssh!

prepare

To allow quick and easy access you need to do prepare the following three things.

install freeRDP on your machine
install XQuartz on MacOS
store the private key material of the EC2 keypair in the SSM parameter store

The first two steps are simple, and will not be explained here. To store the private key material of the EC2 keypair in the SSM parameter store we use the following CloudFormation resource:

  KeyPair:
    Type: AWS::EC2::KeyPair
    Properties:
      KeyName: WindowsServer
      KeyType: rsa

Once you deploy this resource, the private key of the keypair named WindowsServer is stored in the parameter store under the name /ec2/keypair/<key-id>. This is nice, because it standardizes the name of the SSM parameter with the private key material.

start the rdp session

Now we have everything to automate the start of a RDP session, using the following steps.

determine the ec2 instance to connect to
retrieve the private key of the keypair
retrieve the admin password of the Windows server
start the RDP session \o/

determine ec2 instance to connect to

First we determine the EC2 instance id of the machine we want to connect to. In the following snippet, we assume that you have a single machine with tagged with the name.

instance_name=mydemo
instance_id=$(aws ec2 describe-instances \
              --query 'join(`\n`, Reservations[].Instances[].InstanceId)' \
              --output text \
              --filter "Name=tag:Name,Values=$instance_name" \
                       "Name=instance-state-name,Value=running")

retrieve the private key of the keypair

To retrieve the private key of the keypair, we first retrieve the name of the keypair associated with the instance and retrieve the key id.

key_name=$(aws ec2 describe-instances \
          --instance-id $instance_id \
          --query Reservations[0].Instances[0].KeyName \
          --output text)

key_id=$(aws ec2 describe-key-pairs \
         --key-names $key_name \
         --query KeyPairs[0].KeyPairId \
         --output text)

Now we can pull the private key material in:

private_key=$(mktemp)
chmod 0600 $private_key
aws ssm get-parameter --name /ec2/keypair/$key_id \
      --with-decryption --query Parameter.Value \
      --output text > $private_key

retrieve the admin password of the Windows server

To retrieve the admin password of the Windows server, we call get-password-data with the private key.

password=$(aws ec2 get-password-data \
          --priv-launch-key $private_key --instance-id $instance_id \
          --query PasswordData \
          --output text)
rm -f $private_key

start the rdp session

Finally, we have everything to automatically login using RDP. we just have to pick an IP address and run FreeRDP!

ip_address=$(aws ec2 describe-instances \
                --instance-ids $instance_id \
                --query 'join(`\n`, Reservations[].Instances[].PublicIpAddress)' \
                --output text)

xfreerdp /u:administrator /p:$password /v:$ip_address /cert:ignore

That is all there is to it! It is just as easy as running ssh :-p You can find the complete script on github. You can tailor it anyway you like.

Why freeRDP and not Microsoft’s Remote Desktop Client

So you may ask: Why not use Microsoft’s Remote Desktop Client? That is quite easy: it does not support command line options. The alternative would be to generate the RDP file, but on non-Windows platforms you cannot store the password as the required encryption function only works on Windows.

Conclusion

With the freeRDP client, you can fully automate starting an RDP session to a Windows Server running on AWS!

Image by ArtificialOG from Pixabay

The post How to start a RDP session from the command line to a Windows server running on AWS appeared first on Xebia.