Forem: Erik

SIMPLE AWS DEPLOYMENTS ACROSS REGIONS AND ACCOUNTS WITH CDK

Erik — Tue, 07 Feb 2023 10:08:22 +0000

Infrastructure as Code has made it possible to consistently repeat deployments and keep track of the details in version control. AWS CDK, the Cloud Development Kit, is a native Infrastructure as Code solution by AWS that allows you to write you infrastructure definitions in normal programming languages. Deploying a collection of cloud resources with it is as simple as running cdk deploy.

Well, simple... It is simple, as long as your solution spans a single AWS Account and Region. Things tend to become more complicated when multiple regions or other accounts are involved. Regions and accounts are isolated in AWS. Coordinating multi-region or multi-account deployments has often required us to write custom deployment scripts. However, CDK offers us the right set of tools to perform these kinds of deployments in a clear and concise manner.

In this post, we will show how we can use CDK to pick up this tangle of cross-environment deployments and make them simple! By defining our environments either with files in our version control, or with environment variables, we will show a variety of methods. No matter how you want to deploy your code, this post will show you how to do it.

A quick primer on definitions

Two core concepts in CDK are Stacks and Apps. AWS describes a Stack as a "unit of deployment", and an App as "a container for one or more stacks".

A CDK Stack is roughly equivalent to a CloudFormation stack.
It is a template you deploy to a region on an AWS account. A stack can contain many types of AWS resources, but they all exist in the same account and region (with some exceptions for special region-bound resources). So a Stack is the smallest, indivisible unit of deployment.

When we want to deploy a single project across regions or even accounts, we have to deploy multiple Stacks. Using CDK, we have a logical abstraction of a project that consists of multiple stacks: an App. After all, it is a collection of stacks - nothing says it has to be in one region or even in one account.

In CDK lingo, a combination of an account and a region is an Environment.

A cross-environment Application

We will first show how to define an Application that spans multiple Environments.

For the examples in this post, we will be using Python, but the concepts can easily be applied to the other supported languages. The CDK documentation that we will link to shows an example of each individual concept in every supported language.

The AWS Documentation describes the way you can pass arguments to Stack Construct to target an account or region in various programming languages. For demonstration purposes, we will assume we have two different stacks, each of which needs to go to a different account and region. Maybe we are demonstrating a cross-account notification system, with one stack being the sender, and the other being the subscriber in the demo!

The simplest example uses static accounts and regions for its environments:

import aws_cdk as cdk

from demo.event_publisher_stack import EventPublisherStack
from demo.event_receiver_stack import EventReceiverStack

app = cdk.App()
EventPublisherStack(app, "EventPublisherStack",
                    env=cdk.Environment(
                        account='111111111111',
                        region='eu-west-1',
                    ))
EventReceiverStack(app, "EventReceiverStack",
                   env=cdk.Environment(
                       account='222222222222',
                       region='eu-west-1',
                   ))

app.synth()

You can deploy all stacks in this application in one command:

# Option A: deploy each stack in one command
cdk deploy --all

Or with multiple separate commands, which can be beneficial if you need to assume another role for each stack:

# Option B: deploy each stack individually
cdk deploy EventPublisherStack
cdk deploy EventReceiverStack

For a simple demo, using static account ids and regions in your Environments might be good enough. But if this is an Application that will end up in production, you should not hardcode these values.

The CDK documentation has good details on the concepts we will show, though it helps to have a starting point. Because CDK is written in a full programming language, you can use any method you normally would to pass the configuration to it.

To get you started, we have prepared a few short examples showing two approaches you can take.

Method 1: Environment variables

Environment variables provide a lot of flexibility.

import os

import aws_cdk as cdk

from demo.event_publisher_stack import EventPublisherStack
from demo.event_receiver_stack import EventReceiverStack

app = cdk.App()

EventPublisherStack(app, "EventPublisherStack",
                    env=cdk.Environment(
                        account=os.environ['TARGET_ACCOUNT_PUBLISHER'],
                        region=os.environ['TARGET_REGION_PUBLISHER'],
                    ))
EventReceiverStack(app, "EventReceiverStack",
                   env=cdk.Environment(
                       account=os.environ['TARGET_ACCOUNT_RECEIVER'],
                       region=os.environ['TARGET_REGION_RECEIVER'],
                   ))

app.synth()

Where they truly shine is in deployment pipelines. You can define these variables as properties of the target environment.

When deploying from your local command line, a deployment command will likely look like this:

TARGET_ACCOUNT_PUBLISHER=111111111111 TARGET_REGION_PUBLISHER=eu-west-1 TARGET_ACCOUNT_RECEIVER=222222222222 TARGET_REGION_RECEIVER=eu-west-1 pipenv run cdk deploy --all.

Very concise, we know. That's one disadvantage of using environment variables: when run from your local development machine, the deployment commands can become a bit verbose.

Method 2: Configuration files

By putting your environments in configuration files, you make it simpler to see the intended result in the code itself. This does mean putting your account id's in your code. The debate whether this is a security risk has been responded to by AWS (and commented on by others in the community), and this should pose no security risk.

#!/usr/bin/env python3
import json
import logging
import os
import sys
from pathlib import Path

import aws_cdk as cdk

from demo.event_publisher_stack import EventPublisherStack
from demo.event_receiver_stack import EventReceiverStack

log = logging.getLogger("app")

environment = os.environ.get('ENVIRONMENT', None)
if environment is None:
    log.error("Make sure a valid environment is specified in the ENVIRONMENT env var.")
    sys.exit(1)

config_file = Path(f"configs/{environment}.json")
if not config_file.is_file():
    log.error(f"Make sure the config file for the given ENVIRONMENT exists in ./configs/{environment}.json and is readable.")
    sys.exit(1)

with config_file.open() as f:
    config = json.load(f)

app = cdk.App()
EventPublisherStack(app, "EventPublisherStack",
                    env=cdk.Environment(
                        account=config['publisher']['account_id'],
                        region=config['publisher']['region'],
                    ))
EventReceiverStack(app, "EventReceiverStack",
                   env=cdk.Environment(
                       account=config['receiver']['account_id'],
                       region=config['receiver']['region'],
                   ))

app.synth()

We can define our configurations for our deployment stages by placing json files in ./configs. Our developers can define their own configuration file in there, and keep it out of version control if they do not want to clutter the project with their account ids.

> ls ./configs
acceptance.json
dev.json
production.json
test.json

> cat ./configs/test.json
{
  "publisher": {
    "account_id": "111111111111",
    "region": "eu-west-1"
  },
  "receiver": {
    "account_id": "222222222222",
    "region": "eu-west-1"
  }
}

If we then run CONFIG=test cdk deploy --all, our publisher stack will be deployed to account 111111111111, and our receiver stack to account 222222222222, both in region eu-west-1.

An added benefit of this approach is that you can store other configuration in these json files as well. Examples might be the names of artifact buckets or external services that differ depending on the target environment. These can then be passed to the Stacks as extra configuration.

Bootstrapping all your environments for deployment

To deploy to any Environment, that environment must first be bootstrapped by CDK. During bootstrapping, CDK creates the resources it needs to deploy to an environment, such as S3 buckets for artifacts, but also IAM roles and policies that are assumed during deployment. When you run cdk deploy, the CDK command line will attempt to assume one of these roles in the target environment to perform the deployment.

If all your stacks live in the same account, or if you deploy each stack individually, a simple cdk bootstrap in every environment is enough. But if your application spans account, we need a bit more.

Deployments to multiple accounts

If your Application spans accounts, and not just regions, being able to deploy with a cdk deploy --all is a little more involved. You will need to extend trust to assume the CDK IAM resources from one account to another. To do so, modify your cdk bootstrap as follows:

# This will deploy to account 222222222222, so make sure your CLI has a valid login session for that account.
cdk bootstrap aws://222222222222/eu-west-1 --trust 111111111111 --cloudformation-execution-policies "arn:aws:iam::aws:policy/YOUR_CDK_POLICY"

By specifying the trusted account with --trust, the IAM role in account 222222222222 will be modified to be assumed by roles in account 111111111111. If the role or user you assume in account 111111111111 are then allowed to assume this role, you can deploy to the environment of account 1111111111 and account 2222222222 at the same time.

Do note the explicit --cloudformation-execution-policies in this command. When deploying with CDK, the deployment role will assume this policy to perform the deployment. With an unchanged cdk bootstrap, this will default to AdministratorAccess. When bootstrapping an account with a --trust of another account, CDK chooses to make you explicitly choose this role. Using the default of AdministratorAccess carries potential security risks, especially when you allow other accounts to assume this role as well. Create a role with the minimal permissions required to deploy your applications and pass that role to --cloudformation-execution-policies when bootstrapping your environments.

Looking back

CDK offers us powerful abstractions and tools to manage complex applications. These help us truly unlock the promise and power of the cloud. Infrastructure that spans multiple regions is one of these promises, that have been a hard concept to do encapsulate in your Infrastructure as Code for too long a time. By correctly using the capabilities of CDK, we truly unlock this promise.

In this post, we have given you some handholds to get started with these types of deployments. We have shown two approaches to defining your multi-account or -region environments: By using configuration files, we define our environments in a way that's easy to include in our codebase. Alternatively, by using environment variables, we can easily deploy our apps to varying environments.

Of course, there's nothing stopping using another method entirely - that's what having a real programming language to back your Infrastructure as Code definitions enables you to do!

How Infrastructure as Code empowers developers

Erik — Thu, 17 Nov 2022 08:11:01 +0000

Managing infrastructure has been a challenge for a long time. Traditionally, it involved a large amount of manually entered commands, mouse clicks and other human-driven labor.

As more and more companies work with large amounts of servers and duplicated infrastructure across environments, we're running into the limits of this approach. This manual way of working has multiple problems. It is error-prone, as performing actions in a slightly different way can result in large differences. It can be slow, especially when we want multiple copies of each server. Plus, the approach relies on very accurate documentation of the state of each piece of infrastructure, server and procedure. In many cases, this documentation is not kept up-to-date as well as it should be, resulting in problems during development or production.

Infrastructure as Code is all about applying modern practices of software development to the management of your IT infrastructure. It empowers you to do your job as easily as possible. How? By making use of the best aspect of working with computers: automation. It enables consistent rollouts of your software, frequent updates of your servers and it frees up your developers for actual improvements instead of just keeping the existing servers afloat.

This post is intended as an introduction to the concept of Infrastructure as Code. While there are some examples, we will not explain how to write it. We will show various types of Infrastructure as Code that are in active use, and show the ways of working they enable.

Traditional server configuration

Let's start by looking at a very important component of our infrastructure: a server.

In general, a server is just a computer that we run somewhere in a closet or similar to do some serious things. So not your personal laptop or your gaming computer at home, but the ones serving websites, steam downloads, Discord voice calls and so on. Similar to your personal laptop, you install some programs on it when you get it, install updates over time, and add on other programs you want.

Sooner or later, you need to reinstall it - either it's getting too slow, it has malware, or you simply need a backup server. For a PC, it does not matter much if this 2nd installation looks slightly different. But for professional servers, that can lead to nasty surprises. Websites that behave slightly differently, crashes that you had solved on the old installation by changing some obscure configuration. By starting from scratch, we risk forgetting important details.

This means that we need to keep extensive documentation on its installation. Something most people do not like doing.
But it's essential to have consistent installs, and to make a system that others can work with as well.

So we need to find a way to make our setup reproducible, to make it scale.

Repetition, repetition, repetition.

Let's imagine the simplest way we could automate this.

You've gone through the process of setting up a server once.
By keeping careful step-by-step documentation, you described exactly what commands you had to enter to set this server up. Well, then we're already very close to a shell script, aren't we?

echo 'Running setup script...'

# Install the server
sudo yum install mysql-server

# Copy the config file if it does not yet exist
if [ ! -f /etc/my.cnf ]; then
  cp /mnt/usbdrive/my.cnf /etc/my.cnf
fi

systemctl enable mysqld

Every major computer OS has at least one language that you can use out-of-the-box for automation. Maybe it's Bash, maybe it's Powershell, that depends on your operating system of choice. But all of them boil down to being a list of commands that get executed when you run the script.

But of course, we later realize we need some changes in the setup of this server.

It's easy to change the script so a next server looks differently. But you'd prefer to have one central script to execute both for existing servers, that need to be changed, and for new ones. You could solve this by creating a sequential list of scripts. Each one changes the "output" of the last, until we arrive at one central server.

$ ls -lh ./install-scripts
-rw-r--r--  1 34K  Jan 06 16:30 setup-1.sh
-rw-r--r--  1 187B Mar 20 16:30 setup-2.sh
-rw-r--r--  1 20B  May 31 16:30 setup-3.sh

We could go ahead and make a central script that runs all of them. Then we would need to keep track of what the last executed script was on this server. So now we need to keep our own state... This will get tricky quickly.

But!
There's a better way.

---
- name: Install Mysql package
  yum:
    name: mysql-server
    state: installed

- name: Create Mysql configuration file
  template:
    src: my.cnf.j2
    dest: /etc/my.cnf

- name: Start Mysql Service
  service:
    name: mysqld
    state: started
    enabled: yes

This is an example of Ansible, one of many Desired State Configuration tools. The core idea behind desired state configuration like this is that you state what you want the end result to be, not how it should be achieved. The tool, Ansible in this case, will check the current situation, and run the correct commands.

Tools like Ansible even have ways to encapsulate common classes of behaviour in "classes" or "roles". So you can say that a server is "monitored by tool A", and "runs Docker images", and Ansible will apply the correct sets of instructions for both. This means we get to apply some concepts we like to use in programming, like composition, in our infrastructure management!

After converting our commands to desired state in Ansible, we have something we can easily put in version control. Effectively, we've converted the configuration of our server to code.

Deleting everything. Constantly.

Imagine not having to keep a server up and healthy as long as possible. Using modern technologies, it's easy to build your system out of virtual servers that only exist until they need to be changed next. We can create images of a server we've created once, then deploy it as often as we want. If there updates to the operating system or our application, we create a new image, and deploy that.

So we throw away server whenever something needs to change.

What does this look like in practice?

One way is to manually create these images. You can manually boot up a fresh Linux install, go through the motions to set everything up you need except for your application code itself. So you add Nginx, Mysql, a monitoring agent. Then, you create a snapshot of that OS, create servers from it, and all you need to do after that is put your application on it - and done! This is much faster than going through the motion to install all packages from the internet each time, because you effectively only have a single compressed archive you're unpacking - the image.

But we've seen that scripting these steps is easier. There's a very big player in the market that boils down to doing this: Docker Containers.

# syntax=docker/dockerfile:1
FROM node:12-alpine

RUN apk add --no-cache python2 g++ make

WORKDIR /app
COPY . .
RUN yarn install --production

CMD ["node", "src/index.js"]
EXPOSE 3000

We write something that vaguely resembles our shell scripts, and after running it, we get an image that we can deploy as often as we want. It's made to be easy to automate, and you can include almost any setup step in these Dockerfiles.

Where we might have been proud of servers with half a year of uptime, nowadays, virtual machines that don't live longer than a day are not uncommon at all. The main reason for that is not because we hate long uptimes. The real reason these have started existing is to make it possible to scale up rapidly. Being able to create servers in seconds and delete them when you don't need them anymore makes it possible to pay exactly for what you're using, or to run very large jobs every night without paying thousands of dollars a month to keep the servers running constantly.

We call these immutable servers, and the concept in general immutable infrastructure. The alternative we saw before is called mutable infrastructure. Infrastructure as Code is a big enabler of this immutability, and for many solutions, it is easier to maintain than systems we keep tweaking over a period of months or years.

Infrastructure as Code

So far, we've only talked about servers. Infrastructure is, of course, a lot more than just this. We also have networking, storage, the underlying hardware of the server, and so on.

In practice, any component that is virtual (as opposed to, say, a physical you have to plug in by hand) can be automated, and all the major Cloud providers have virtualized almost anything you can think of. And it's here, in the cloud, that Infrastructure as Code - true infrastructure, not just servers - truly starts to shine.

When you first start out working with a cloud provider, you might log on to their web console, launch a virtual machine or two, create a managed databases and connecting them.
But then, you need to do it again for your production environment, and you have to make the exact same clicks so the environments are similar enough.

But you can automate this.

Cloud providers have done something awesome: they created and published API's with which you can do nearly anything you can do manually in their web consoles.

One approach would be to write another shell script. This time, it runs a series of HTTPS calls to your cloud provider.
But: we've learnt about the power of desired state configuration. So we know there is something better.

Resources:
  StaticFilesBucket:
    Type: AWS::S3::Bucket

  ServerlessWebHandler:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.9
      CodeUri: s3://my-example-bucket/serverless-web-handler/v1.zip
      Environment:
        STATIC_FILES_BUCKET: !Ref StaticFilesBucket

This is Cloudformation. It's the official language of Amazon Webservices, to define our cloud infrastructure as desired state.

We can put this in a GitHub repository and automatically deploy it to the correct AWS account - test or production - at the push of a button. You change your code, automated checks verify it, someone looks if it's functionally, you merge it. You run your CI/CD pipeline, and minutes later your AWS account is up-to-date with the new environment.
It created virtualized, managed databases, message queues, networks and servers based on a bit of configuration.
And after testing it, you click another button, and it replaces your production environment in-place, without downtime.

So there you have it: we can now create complete, complicated, multi-tier application infrastructure as YAML.

Infrastructure as real Code

If you're like me, you might feel a bit deceived about hearing "code" and seeing YAML. The good news is, we have much better alternatives nowadays. For most Cloud Providers, there are development kits that allow you to write normal code - like Python, Java or C# - and generate your infrastructure from that!

from aws_cdk import Stack, aws_lambda, aws_s3
from constructs import Construct

class MyExampleStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs):
        static_files_bucket = aws_s3.Bucket(
            self,
            id="StaticFilesBucket",
        )

        serverless_web_handler = aws_lambda.Function(
            self,
            id="ServerlessWebHandler",
            runtime=aws_lambda.Runtime.PYTHON_3_9,
            code=aws_lambda.Code.fromAsset('./functions/web-handler/'),
            environment={ "STATIC_FILES_BUCKET": static_files_bucket.bucketName }
        )

        static_files_bucket.grantRead(serverless_web_handler)

What you see here is CDK: the AWS Cloud Development Kit. CDK is a collection of libraries and tools you can use to create cloud infrastructures. It's available in multiple languages - so far in Javascript, Typescript, Python, Java, C# and Go.

It can generate all the same infrastructure you can with CloudFormation. But instead of plain old YAML, you can use a real programming language.

YAML has some distinct disadvantages compared to a real programming language. The lack of custom functions, for/while loops, easily reusable libraries and much more. By putting our infrastructure definition in actual code, the intent of our definition is much more readable.

By "synthesizing", or running, a CDK project, the underlying CloudFormation will be generated. In that way, the step from CloudFormation to CDK feels similar to other steps forward we've made from low-level programming languages to high-level ones! But in this case, we still often run into the "Assembly" variant of the code: CloudFormation. That makes it a good idea to check the CloudFormation that is generated by your CDK code.

At the time of writing, we highly recommend anyone getting started with Infrastructure as Code on AWS to try out CDK.

The power of the automated cloud

So far, we've seen a few steps of automation. From shell scripts, to desired state configuration of long-living servers, to immutable servers, and on to complete immutable infrastructures defined as desired state. But you may still be wondering - what's all the fuss about?

So let's take a little more time to envision something that becomes possible with the automated cloud.

Imagine a company running a couple of webservices that are used by customers to send large sets of data and to retrieve analyzed results. They're already running these in Docker containers - which means that, because creating the "server" these services run on is automated, we can easily recreate these in various environments.

We create an AWS account for each environment - test, acceptance and production - and spin up each service in each environment. Changing the "size" of each underlying resource, such as CPU, memory and storage, is a matter of passing along different parameters in a few fields of our Infrastructure as Code. That makes it incredibly easy to have otherwise identical services in each account, with differing amount of compute resources. This makes it easy to save money in the test environment. One step further would be to use _serverless services, where we only pay for what we use - but that's a different topic!_

The services can place the submitted datasets in a managed storage location for processing.

Next, we have a few worker processes that pick up this data and generate the information our customers want from us.
Thanks to the ease with which we can automatically create and delete servers, we actually scale the number of workers up and down depending on the amount of work to be done.
This means we can run cheap when it's calm, and still work very fast during the busier days.

We now have an autoscaling environment with barely any maintenance to the underlying system. Thanks to the automated cloud. But it gets better.

We can create an AWS account for each developer, which means they can all work completely isolated from each other when they want. And after each merge to our integration branch - after a developer finished a bug or feature - we can automatically test the code in a real environment. We deploy the entire stack to another AWS account, and run automated tests against it as if we were our customers. After finishing up the test run, we delete all the resources again, and barely spent any money for a true end-to-end test.

All the while, the developers were focused on what they enjoy doing - building things - instead of keeping the systems running, tweaking the configuration of our message queues, adding and removing storage, etcetera. Setups like these have been possible for a long time. But we've only recently arrived at a point where it's become feasible for companies of any size to implement this, without spending much more money on resources or extra developers.

And none of this is hypothetical.
We've seen and helped our own customers create setups just like this. This is actually used - a lot - in production, and while it will take some getting used to, it will speed up your development enormously.

Good Infrastructure as Code empowers you, as a developer, and as a company.