Forem: Thomas Poignant

Create a bastion with AWS CDK

Thomas Poignant — Mon, 27 Jan 2025 08:15:38 +0000

A bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet. With the public cloud era, this is one of the favorite ways to access your private resources.

Let’s say you want to SSH your EC2 instance in your private subnet you can use the bastion which is in a public subnet to forward your traffic to your EC2 without to open public access to the EC2.

How to manage your SSH key

Managing your SSH keys in the bastion is the worst, it’s fine for a one-person project, as soon as you are a team with people leaving and some arriving you end up adding/removing keys all the time.

For this typical use case AWS has created a service called AWS Systems Manager Session Manager and as they said :

Session Manager provides secure and auditable instance management without the need to open inbound ports, maintain bastion hosts, or manage SSH keys.

You will ask me why creating a bastion if we have Session Manager?
A classic answer is to query a database located in a private subnet.

How to build our bastion with CDK

AWS Cloud Development Kit (CDK) lets you create your infra using real languages. In this example, I will use python but CDK is also available for java, javascript, and typescript.

The good thing when you want to create a bastion with CDK is that there is a construct for that:https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-ec2.BastionHostLinux.html

This creates a linux bastion host you can use to connect to other instances or services in your VPC.

The recommended way to connect to the bastion host is by using AWS Systems Manager Session Manager.
The operating system is Amazon Linux 2 with the latest SSM agent installed.

The file below will create a bastion with a security group associated.

https://gist.github.com/thomaspoignant/b498e45f2f4c939e53e429ae9cdced39#file-bastion-py

So as you can see, first, we are creating a security group with no inbound port open (it means that the machine cannot be SSH from anywhere). After we are creating the bastion with a few parameters like the instance type, and where to start it (here in a public subnet) and we are setting the security group to the instance.

Deploy the CDK stack

Now that our bastion configuration is ready, we just need to deploy the stack with:

cdk deploy

Because we are deploying some changes to a security group, CDK will prompt you to validate the changes.

Wait till your stack is done. What it does is running a cloudformation in your AWS account and the command line gives you some information about what was built.

And if you are going to the console you will be able to see that your new instance has started.

Connect to the bastion

To connect to the bastion you need to have your aws cli setup and your credentials stored.

The first thing to do is to get the InstanceId of your bastion. For that you can run:

export AWS_REGION=eu-west-3
export BASTION_INSTANCE_ID=$(aws ec2 describe-instances \
                          --region=$AWS_REGION \
                          --filter "Name=tag:Name,Values=my-bastion" \
                          --query "Reservations[].Instances[?State.Name == 'running'].InstanceId[]" \
                          --output text)

Now you have an environment variable called BASTION_INSTANCE_ID with the InstanceId of your bastion.

Now we can use AWS Session manager to access to the bastion:

aws ssm start-session --target $BASTION_INSTANCE_ID --region=$AWS_REGION

By running this command every user of your account with the right IAM access will be able to connect to the instance. No SSH key to manage anymore, all the access is given via the IAM policies.

Conclusion

With less than 40 lines of code, you can create a bastion with Session Manager using AWS CDK.

All the complexity is hidden by the high-level construct of CDK, it means that you can be focused on what you want and not how to do it.

I really ❤ CDK

🚀 Announcing GO Feature Flag v1.0.0, get more from your feature flags

Thomas Poignant — Fri, 03 Feb 2023 14:47:19 +0000

🚀 Announcing GO Feature Flag v1.0.0, get more from your feature flags

We are so happy to finally announce that GO Feature Flag v1.0.0 is released. 🎉 🥳 🚀

It was a long journey before this release but we are happy to propose you an even better experience with feature flags always keeping simplicity in mind and building a lightweight solution.

Our goal is still the same, we want you to experience the world of feature flags with a lightweight self-hosted solution.

With this new version you will now be able to enhance your usage of feature flagging with opening the creation of more complex and more advance toggles.

What is changing?

Flag configuration format

The main change is the format of your flags configuration.

After using the version v0.x.x for quite some time we notice some limitation on the way our flag configuration was designed.
It was hard to return more than 2 different values with the same flag and we wanted to find a better way to represent our flags to be more extensible.

The new representation of the flag is a bit more verbose but allows a lot more possibility in the long term, being able to create more advanced flag.

Old format:

In the previous versions the flag representation was looking like this:

test-flag:
  rule: key eq "random-key"
  percentage: 40
  true: true
  false: false
  default: false

As you can see on this example you can only have a value for the true variation and for the false variation, the defaultvariation was applying if the user was not part of the rule.

New format:

The same flag in the new format will looks like this:

test-flag:
  variations:
    variation_A: true
    variation_B: false
  targeting:
    - query: key eq "random-key"
      percentage:
        variation_B: 60
        variation_A: 40
  defaultRule:
    variation: variation_B

With the new format you have an unlimited number of variation possible you can define but also more than one rule for a specific flag.

It allows to target different types of users with different values for the same feature flag.

Let’s take a real life example, we want to get the background color for our website based on this criterias:

pro users should have a red background
enterprise users should have a green background
40% of the free users should have a grey background and 60% a yellow background
other types of user have a white background

We can come-up with a flag, the configuration will look like this:

user-background-color:
  variations:
    pro: #ff1f00
    enterprise: #114f03
    free_grey: #b0b3af
    free_yellow: #ecec0a
    default: #ffffff
  targeting:
    - query: type eq "pro"
      variation: pro
    - query: type eq "enterprise"
      variation: enterprise
    - query: type eq "free"
      percentage:
        free_grey: 40
        free_yellow: 60
  defaultRule:
    variation: default

As you can see we can have a way more advanced flag configuration with this new format. This format is solving all the limitation we had with the version v0.x.x of GO Feature Flag.

Migrate from v0.x.x to v1.x.x

First of all you should know that all flags from the version v0.x.x are still compatible with this new version of GO Feature Flag. We made it possible to reduce the disagreement to have to convert all your flag directly.

BUT we encourage you to migrate your flag to the new version and for this we have created a tool called go-feature-flag-migration-cli to convert flags in v0.x.x format to v1.0.0 format.

The easiest way to use it is probably to use our docker image like this:

docker run \
  -v $(pwd)/your/configuration_folder:/config \
  thomaspoignant/go-feature-flag-migration-cli:latest \
  --input-format=yaml \
  --input-file=/config/my-go-feature-flag-config-v0.x.x.yaml \
  --output-format=yaml \
  --output-file=/config/my-go-feature-flag-config-v1.x.x.yaml

Documentation

We use the opportunity of the v1.0.0 to revamp our documentation and to highlight how you can use GO Feature Flag in 2 different modes.

The GO module, this is where we started and this is the core of what we propose. In the documentation you will find all the information on how to use the module into your GO project.
Using Open-Feature, the new open-source vendor agnostic standard for feature flags. You can use the relay-proxy our server component that allows to use GO Feature Flag with multiple languages (for now we support javascript, typescript, java, GO and .Net). In the documentation you will find how to use the GO Feature Flag providers in combination with the Open Feature SDKs.

Flag Editor

Our old flag editor was not compatible anymore with the new v1.0.0 flag format.

So we decided to restart our flag editor from scratch and to allow the possibility to create flags configuration from a simple UI.

For now it is a tiny UI that make your life easier, but the goal is to evolve it more and more to be the place where you want to go to build your new feature flags.

What has not changed?

Yes a lot of things have changed but our internals are solids and we have ensure to be consistent with the previous versions.

Our main points of attention where the following:

User segmentation is similar between v0.x.x and v1.0.0 meaning that if a user was affected by a flag (in a percentage for example) he will continue to be in the same cohort.
All retrievers and data exporters have not changed and are still working exactly the same way as before.
You can continue using GO Feature Flags with your old flags.
Moving to the version v1.0.0 is transparent and you just have to change the version.
You can still use our advance rollout capabilities such as experimentation, scheduled rollout, progressive rollout …

Feedback

Feedbacks is our only way to evolve GO Feature Flag, so please contact us to give us any feedback.

Via email
Via a GitHub Issue

10 things about AWS CDK

Thomas Poignant — Thu, 02 Feb 2023 08:20:36 +0000

Some months ago I joined a new team, they were on AWS and using cloudformation for some stuff and, a lot of changes were made directly in the console.

After some discussion we’ve decided to start fresh on a new infrastructure code, with some constraint in mind:

We are not an OPS team but we are managing the infrastructure by our self.
We want no manual action in the AWS console.
The languages we are using in the team are GO for the backend and scala/python for data.

We have considered to re-work with Cloudformation and also with Terraform, but since no-one wanted to written thousands of lines of YAML we’ve decided to use AWS CDK.

After all this journey of building our new stack, I can give you these 10 things to know before starting a CDK project (but I can already tell you that this is awesome).

1. Write code in a real language, not YAML

You probably know that if you have ever looked to CDK, but despite other infra-as-code technology you can write your stack in real language.
Why this is better?

You can keep your favorite tools to monitor your code, such as your IDE, your linters, quality checks …
Describe your infra using an object model.
You can do unit test your code. (This one is the awesome part 🙌 🙌 🙌)

2. CDK support multi-language based on JSII

JSII is a framework developed by the technology that enables the CDK to deliver polyglot libraries from a single codebase!

https://github.com/aws/jsii

Currently AWS CDK is supporting:

Typescript / Javascript
Java
Python
.NET
GO

This is awesome you can choose your favorite language in the list and start building your stack.

⚠️ This is not that easy!
In my team, since there is no GO support in CDK we’ve decided to use Python because this is our second language.
To be honest, this is not a bad choice everything works fine but it has some limits of not using Typescript/Javascript.

Why typescript / javascript is better?

All components are written with typescript at first, so you are sure that they will work for you. With other languages, it works perfectly if you are using default CDK constructs but as soon as you are using 3rd party constructs it less certain that it will work.

And you can also have some weird documentation problems (see issues below).

3. CDK hides complexity

Despite cloudformation or terraform where you have to write everything you want to have, CDK provides high-level constructs that do the work for you.

For example, if you want to create a VPC with a set of subnets and associate Nat Gateways it can end up with a thousand lines of code CF or TF.

With CDK most of the stuff is done automatically for you and you, as you can see with the code below.

subnet_configuration[0] = ec2.SubnetConfiguration(
    name='Public',
    subnet_type=ec2.SubnetType['PUBLIC'],
    cidr_mask=21
)subnet_configuration[1] = ec2.SubnetConfiguration(
    name=name,
    subnet_type=ec2.SubnetType['PRIVATE'],
    cidr_mask=21
)vpc = ec2.Vpc(scope=self,
              id='test-vpc',
              subnet_configuration=subnet_configuration,
              max_azs=3,
              cidr='10.0.0.0/16',
              nat_gateway_subnets='Public',
              enable_dns_hostnames=True,
              enable_dns_support=True,
              )

4. CDK is based on cloud formation

This one is really important to understand, CDK outputs some cloudformation code and apply it to your AWS account. So if something is not available in cloudformation it means that there is no chance you will be able to do it with CDK.

But if like us you had a cloudformation stack before or if you want to import a 3rd party cloudformation script, you can do it.

# This code use a datadog CF template in a CDK stack.
cfn.CfnStack(
    scope=self,
    id='PolicyMacroStack',
    template_url='https://datadog-cloudformation-template.s3.amazonaws.com/aws/datadog_policy_macro.yaml'
)

5. 🌬 CDK has a fast release cycle

I’ve started my CDK project 4 months ago using version v1.42.0 of CDK.
The actual version (when I write this article) is v1.74.0, which means that I have made 32 version updates on my actual stack.

So be prepared to upgrade your dependencies all the time.

6. Experimental higher-level constructs

Related to point 5, be warned that if you are using Experimental constructs you can expect some breaking changes.

Experimental: Higher level constructs in this module that are marked as experimental are under active development. They are subject to non-backward compatible changes or removal in any future version. These are not subject to the Semantic Versioning model and breaking changes will be announced in the release notes. This means that while you may use them, you may need to update your source code when upgrading to a newer version of this package.

The good thing is that for every construct available (even if they are experimental) you still have access to CFN Resources which is basically a wrapper over cloudformation and these resources are always stable and safe to use.

7. Fine-grained assertions are only available for Typescript

As I said before you can unit-test your infrastructure. That being said if you want to use the full power of testing with CDK, you are kind of stuck using javascript or typescript because you can use the fine-grained assertions framework.

Fine-grained assertions test specific aspects of the generated AWS CloudFormation template, such as “this resource has this property with this value.”
These tests help when you’re developing new features, since any code you add will cause your snapshot test to fail even if existing features still work. When this happens, your fine-grained tests will reassure you that the existing functionality is unaffected.

These fine-grained assertions are awesome because this makes it easy to test for real your expected stack.

https://docs.aws.amazon.com/cdk/latest/guide/testing.html

8. Nested stack vs Construct

If you have a lot of components in your stack at some point the question will be how to structure it.

You can use a lot of classes that extend Construct this the way most people are doing because before last week it was impossible to run a proper cdk diff to see what will be changed in your stack before deploying it.

The problem with using Construct is that you can reach the 460,800 bytes limit from CloudFormation, and you can reach this limit quickly because CDK is doing a lot of stuff for you behind the scene.

So in that case you can use some Nested Stacks, the problem was that when you were using nested stacks you were totally blind before applying changes. But last week Cloudformation has announced that changesets now support nested stacks.
So if you want to split your stack, please use nested stacks now.

9. CDK is multicloud

CDK is a project from AWS, but now you can use it as a multi-cloud tool since Hashicorp has released CDK for terraforming.

What it changes is that your CDK build will output some TF files, so it can apply in any cloud provider because terraform supports most of them.
CDK for Terraform: Enabling Python & TypeScript Support
Today, we are pleased to announce the community preview of the Cloud Development Kit for Terraform, a collaboration…

https://www.hashicorp.com

As you can see this project is really new, but it gives hope in the future of CDK and fingers crossed🤞 that it will become a standard even outside of AWS.

10. CDK everywhere (CDK8s, projen …)

CDK is such a good idea that eladb (the creator of CDK) uses the CDK philosophy everywhere.

cdk8s is in beta and this is CDK for kubernetes, it is AWS agnostic and can be used on any K8S project.
Even if this is a beta, you can already use it in production, the only visible tradeoff is that all kubernetes features are not implemented yet.

eladb has also a side project called projen that allows you to create typescript/javascript project with a init script that maintains your config files (package.json, tsconfig.json, .gitignore …).

Conclusion

I don’t regret to decide to use CDK instead of other technologies, it makes the work WAY easier and all the team understands what we are doing (not only the ops part of the team like before).

This is a work in progress tools, but almost all major AWS component are already well covered so you don’t have to worry when you start building your stack and with less than 10k lines of codes we have built a complete infra where we managed a lot of components (VPC, subnets, security groups, EKS, Elasticsearch, secrets, IAM, monitoring …).

My advice is to test CDK if you want a build a new stack, you will be convinced by your self after a few hours of using it.

AWS CDK Continuous Integration and Delivery Using Travis CI

Thomas Poignant — Thu, 03 Jun 2021 16:01:06 +0000

Ship infrastructure like you ship your code

AWS Cloud Development Kit, better known as CDK, is changing the way we do infrastructure as code by using a proper programming language to build your infra.

Since we are building infra with dev tools, we should have a CI/CD for our CDK stack code that looks like any other project. If you look online, AWS explains how to do CI/CD with code build/code pipeline, but these probably aren’t the tools you are using in your day-to-day.

In this article, I will present to you the deployment pipeline I’m using to deploy my CDK stack based on GitHub and TravisCI.

Continuous Integration

Continuous integration looks like any code project CI. We validate our code, run the tests, and try to see if the build failed.

This is run in every commit to ensure that we have the shortest feedback loop possible.

So in Travis configuration, it will look like something like below (here, we have a CDK Python stack):

This CI is classic. We lint the code, test it, and export the coverage. We run cdk synth for each stage on each commit to be sure that the config does not break one stage in particular.

As you can see, we also use multiple Travis CI stages to run everything in parallel in order to have the shortest feedback loop possible.

Open Pull Requests to Validate Your Deployment

The best strategy I have found to deploy the code to AWS is to have one branch per stage (dev, pre, pro), so every time a new commit reaches one of these branches, we are deploying the actual stack. Yes, it looks like GitOps!

But we don’t want to blindly deploy our CDK stack on an AWS account. So I have an automatic way of opening pull requests when something is merged to the main branch:

Each time something is merged to the main branch, we have a specific job that runs cdk diff -c stage=<STAGE> and opens a pull request to the destination branch.

For that, we use the script below. As you can see, it runs cdk diff and uses the results to open a pull request using the GitHub CLI:

Now we have to call this script inside Travis, so our .travis.yaml file will look like this:

As you can see, we are now using some env variables to specify our AWS Credentials in addition to our GitHub Token. Please make sure to encrypt them before committing them in your repo.

Continuous Deployment

Now that you have your pull requests open for all your stages, it’s time to deploy the changes to your stack!

On your PR comment, you are able to see the diff, so you will no longer be blind when it’s time to deploy a specific stage.

The next step for the CD is to deploy the stack. Each time we have a change in one of the release branches, we want to run cdk deploy -c stage=<STAGE>.

The complete .travis.yaml file will look like this. As you can see, each stage uses a condition for the deployment to be sure to apply it only to a specific AWS stage:

Conclusion

With this GitOps approach, it is easy to understand what you’re deploying in your stack and all your changes are in Git.

It also helps engineers with less knowledge in CDK to feel confident in the deployment of the infrastructure because no manual steps are required to deploy to a “real” AWS stage. The main advantage is that you are using the same tooling that you are using for the rest of your projects.

Golang: Build and Deploy an AWS Lambda using CDK 🚀

Thomas Poignant — Wed, 26 May 2021 15:21:18 +0000

If you read my previous posts you can see that I am working with Golang and AWS a lot, so in this one, I will present how I deploy a serverless API in AWS Lambda with an API Gateway to access it.

I will write the API using GO and go-chi/chi to handle the http routing layer.

TL;DR

All the sources are available in thomaspoignant/cdk-golang-lambda-deployment repository.

thomaspoignant / cdk-golang-lambda-deployment

Create a serverless API in GO

So the first things we will do is creating an API using go-chi and the aws-lambda-go SDK, so we need to import these dependencies:

go get github.com/aws/aws-lambda-go
go get github.com/awslabs/aws-lambda-go-api-proxygo get github.com/go-chi/chi
go get github.com/go-chi/render

You can see that we’ve also imported github.com/awslabs/aws-lambda-go-api-proxy this framework allows using go-chi with a lambda handler.

Now we can create our main.go file that contains the logic of our API.

In this file, we have 2 important functions, main() and handler().

This is important to understand that the main() function is called only when the lambda is starting, so don’t do anything in the main() if you want that action to happen for each API request.

The handler() function is where the lambda is called, so you will go on this function every time a request comes in, and since we are using a chiadapter the handler will use the go-chi router to handle the requests.

We are using this adapter to keep all the benefits from a framework to handle the http layer, it allows us to have better routing, but also to use some fancy helper to get your params, or write your responses (in the repo we are using go-chi-render for that).

👇You can check the repo to see the complete code of the lambda.👇

thomaspoignant / cdk-golang-lambda-deployment

Build & Deploy our lambda with CDK

Now that our lambda function is ready we will have to build it and deploy it.

So our CDK stack will contains these steps :

Create the GO binary we will use in our lambda
Deploy the code as a lambda function
Deploy an AWS API Gateway to access this API on the internet.

Create GO binary and deploy it

Since we are using CDK it is to serve and build the lambda inside our infra as code.

So as you can see the lambda.Function object is taking 3 parameters:

runtime: information on the language of the lambda (here we are using GO)
handler: the name of your go binary
code: the code to run in the lambda, as you can see we are using lambda.Code.fromAsset this takes the code location and create the binary from the code. To execute the build, CDK is launching a docker image (you need docker on your machine) and use the makefile to build the binary with no dependency of the current machine running CDK.

Add an API Gateway to access the lambda

Since lambda functions are not accessible directly through HTTP, let’s add an API Gateway in front of our lambda function.

Doing this is super easy with CDK, this is only one object creation call to LambdaRestApi with the lambda you just created and a description.

It will do everything for you under the hood, by creating a /{proxy+} route on the API Gateway and the associated deployment, it means that all the routes will go directly to lambda function and you have to manage the routing on your code, and this is EXACTLY what we want to do!

Conclusion

As you can see with not much code we built/deployed and exposed a serverless API.

You can check the full repo here, and use this as an example to start building a more complex logic in your API.

It is also super easy to manage more than 1 lambda function in your stack, you just have to call multiple time the CDK functions so enjoy ✌️

thomaspoignant / cdk-golang-lambda-deployment

cdk-golang-lambda-deployment

This repo is an example on how to deploy a serverless API on AWS Lambda using aws CDK.

Associated article https://faun.pub/golang-build-and-deploy-an-aws-lambda-using-cdk-b484fe99304b

Lambda

The GO Lambda function is a simple API using go-chi/chi to handle http requests.
The code is located in the /lambda/api folder.

Deployment with CDK

The CDK stack is also simple, it build the binary of the lambda and deploy it with an API Gateway in front of this lambda to access it through HTTP.
The code is located in the /infra-cdk folder.

View on GitHub