Forem: Custom Ink Technology

Custom Ink's Kubernetes Journey

Martin Bonica — Wed, 22 Nov 2023 16:54:04 +0000

This is a crosspost from Custom Ink's tech blog.

There's been an elephant in the room during the past few Custom Ink Tech Blog updates. Perhaps it's been alluded to, but we've been consistently putting off addressing it here, simply because of its scale, as well as how many subsequent blog posts could be written wrestling with its implications. (This, of course, means we've also been putting off said subsequent blog posts, because we never got around to writing this one.)

So, it's time to rip off the proverbial band-aid; the majority of Custom Ink's compute workload is now running on Kubernetes, by way of Amazon EKS. (The rest is, as you might have guessed by reading this blog, on AWS Lambda.) We no longer use Chef, and we no longer use Capistrano to deploy our services and applications directly to EC2 instances. We run everything behind Customink.com without keeping track of individual servers and persistent file systems, and all app-specific infrastructure configuration - OS, libraries, packages, resources allocated, ingress - now lives in the same Git repository as a service's source code, where developers can change them as they see fit.

The Journey Begins

Even before Custom Ink moved to the cloud, the pattern of persistent, stateful servers managed by Chef with code deployed by Capistrano served us well. Our infrastructure wasn't exactly ephemeral; it would take some manual action to get a new server up, get a load balancer pointed at it, and tell Chef to tell Capistrano it was ready to deploy to. Servers tended to stick around for a while (so, not ephemeral), and when it was time for an upgrade, we'd upgrade them (so, not immutable either). That said, that pattern stayed with us in the cloud, followed us to a resilient multi-AZ layout, and absolutely beats doing everything by hand.

In this engineer's opinion, a cultural shift within development is what motivated us to begin the journey to ephemeral and immutable infrastructure. As you might have gleaned from some past blog posts, Tech Inkers know a thing or two about Heroku. The venerable PaaS taught a generation of developers that, if they could generate their runtime environment and compute resources from code instead of asking a sysadmin for it, they could deliver more, and do it faster. Over the course of time, everyone arrived at the same conclusion; we can do this faster, and with less work.

It's important to note the moment in which this sea change took place. These conversations were happening at Custom Ink in late 2019. (Yes, this blog post has been marinating for quite some time.) This mean that the immutable, ephemeral platform we were already dipping our toes into - AWS Lambda - was still a little rough around the edges. We were still getting a hang of how to use lambda layers to bring in binaries we needed for some of our services (say, MySQL and Oracle connectors) into the execution environment. Secure secret injection out of the box was not really a thing yet.. Finally, keep in mind that Lambda would not support OCI images for another year. It was (and still is) fertile ground for new development, but we had a backlog of existing services that needed a new home.

OCI images (Docker, for the layperson) and a container orchestration platform seemed to be our quickest way to lift and shift our services to an ephemeral, immutable infrastructure. Thankfully, our applications were already stateless; environment-specific configuration was handled by dotenv, which was happy to ingest values from environment variables, and everything else lived in either databases, memory caches, or S3 buckets. There were few to no persistent files to worry about. Thanks, Rails!

Even in 2019, there were some options for running containers in AWS. Kubernetes won because it had mindshare on the team, and because it was the most flexible. (This last part, as we will soon learn, can be a blessing or a curse.)

If you're curious, our first customer-facing production service hosted on Kubernetes was the international checkout component of our website, which launched in March of 2020. The web frontend of customink.com was on Kubernetes in June of 2022, and the last to make the leap was the service that handles clipart in the design lab, which moved in January of 2023.

EKS Essentials

A familiar feeling to anyone who has spun up an EKS cluster: looking at the console, maybe creating a "hello world" pod and seeing it schedule somewhere, and thinking "OK, what now?".

Although Fargate, EKS managed node groups, and Karpenter have since come along and made things a bit easier, there's a hard line between resources managed in AWS API and resources behind the API of your Kubernetes cluster. If you want to spin up a pod, you need to connect to the control plane of your EKS cluster and ask for it. That means you have to authenticate to it first. If you want something on the internet to be able to connect to that pod, you need to figure out how to get Kubernetes to tell AWS to point an IP address (or two, or three) at a port where your pod is reachable. If you need to get secrets (say, API tokens or passwords) into the environment of your pod, it's on you to decide how to get that out of your secrets manager of choice. EKS is very hands-off in that regard, unlike other, more opinionated Kubernetes distributions. This is how we chose to instrument our Kubernetes clusters; it is not the only way to do this, and there's certainly more CRDs and daemonsets running than those we listed here, but mastering these were critical to our adoption of Kubernetes.

aws-auth

One nice thing EKS comes with out-of-the-box is the AWS IAM Authenticator for Kubernetes. It allows us to map IAM principals to users within Kubernetes. This is handy; it means we don't have to worry about giving everyone a Kubernetes identity, and instead just scope their privileges based on their IAM identity.

By default, the only user who is allowed to talk to the Kubernetes control plane is the IAM principal that created the cluster. Even if that principal was a role assumed by a group of people, that's less than ideal; after all, from Kubernetes' perspective, that means there's only one user, and no way to reduce the scope of that user's access (which is, of course, full admin). Thankfully, the IAM authenticator gives us one more thing; the ability to edit a ConfigMap called aws-auth to associate IAM principals with groups, and groups with Roles or ClusterRoles. This means we can have an IAM role called "ReadOnly" that we map to a Kubernetes role that can only do "get" actions, or an IAM role called "PowerUser" that's allowed to restart deployments - you get the picture. This is how we map external entities, such as engineers, developers, or external software that operates on the Kubernetes API, to Kubernetes RBAC. All the calling entities need to do is run or implement the AWS IAM Authenticator process, which in a nutshell uses IAM credentials to call the AWS EKS API. The caller gets a temporary session token in return, which is then good for talking to the Kubernetes control plane.

View on mermaid.live

Ingress

As we've written before, we make use of path-based routing for some (but not all) of our customer-facing applications. Thankfully, making this change on the Kubernetes side was not a problem. The Kubernetes Ingress API allows for rule-based routing, and the AWS Load Balancer Controller allows us to implement the ingress spec by creating and controlling Application Load Balancers. We use the AWS Load Balancer to deploy a new IngressClass - let's called it the "shared ingress" - which represents a load balancer with no rules, just various ACM certificates and security settings. Each individual application then is responsible for declaring an ingress, specifying what paths and hostnames they listen on, using using the aforementioned shared ingress as the IngressClass. The AWS Load Balancer Controller sees that an Ingress is using an IngressClass it created, and adds the corresponding rules to the "shared" load balancer. This allows us to control load balancers however we want, and even share them, from within the Kubernetes API.

External Secrets

There are a lot of ways and places to manage secrets, but our biggest requirement is for secrets to be encrypted at rest, restricted by IAM policies, and managed somewhere other than the Kubernetes control plane. At this time, we are using SecureString parameters in AWS Systems Manager Parameter Store. The parameters are encrypted with KMS, and IAM limits the principals that can decrypt them.

We elected to use the External Secrets Operator as a mechanism to turn SSM parameters into Kubernetes Secret resources. This tool provides a number of Kubernetes Custom Resource Definitions (that is, new objects for the API to use) which allow us to access external secret stores. The key objects here are SecretStore and ExternalSecret resources. The SecretStore object instructs the ESO to connect to a secrets backend of choice. An ExternalSecret object can then be created, referencing the SecretStore we created, asking for certain values to be pulled from that backend. (In our case, the SecretStore is pointed at AWS SSM.) When an ExternalSecret is created or modified, the ESO will connect to the secrets backend, retrieve the secrets, and create a Kubernetes Secret object populated with the values. We can them use this Secret as we would any other Secret or ConfigMap, mounting it on a pod's filesystem or exposing it as environment variables to our application can use the secrets.

View on mermaid.live

Service Accounts and IAM

As you might have inferred by now, there's lots of pods within our Kubernetes cluster that need IAM permissions; either to perform infrastructure actions to support our services (create ALBs, get secrets, etc) or in the normal course of a service operating (say, we're uploading a file to S3 or sending a message down an SQS queue). While permissions associated with the IAM role of a Kubernetes worker node can be inherited by all the pods on them, we really don't want that; after all, that would mean everything on the cluster would have the same role, with very broad access. We would prefer that applications running in Kubernetes be scoped to their own IAM role. Thanks to the OIDC capabilities of EKS and IAM, this is pretty easy to do.

EKS clusters come with their own unique OIDC URLs. These URLs, along with a unique thumbprint associated with them, allow IAM OIDC providers to verify that a caller from outside of AWS (say, the EKS control plane) is who it claims to be. We can then add the OIDC provider for our new Kubernetes cluster to the trust relationship of an IAM role. (We can scope the trust relationship further, to namespace and service account name, if we really don't want the app that uploads clipart to be able to assume the role that lets it play with load balancers.)

With this in place, all we need to do is create a ServiceAccount object, add the IAM role we want to assume as an annotation, and associate it with our pods.
Some more EKS-specific magic then takes place in the background; the AWS EKS Pod Identity Webhook sees that a pod is associated with a ServiceAccount that wants to use an IAM role, and makes an AWS STS call to IAM asking for a temporary web identity token for that role. If OIDC was set up right, it gets a valid web identity token, and mounts it, along with the name of the region it's running in, as environment variables in the pod. At that point, your code, and the AWS SDK (assuming you're on a version new enough to recognize web identity tokens) will run as if it's on an EC2 instance with an IAM role.

This way, we can associate pods with different IAM roles, and prevent services from encroaching on eachother's permissions.

View on mermaid.live

The Deployment Pipeline

View on mermaid.live

Early on, we elected to come up with a solution that was CI tooling-agnostic; whatever we did would not be bound to Jenkins, CircleCI, TravisCI, GitHub Actions, or the crontab running on the gaming PC in the closet. We didn't want to anchor ourselves to any of those things. Instead, we packaged all of the various scripts needed for our deployment process into a Ruby package we refer to internally as "KTool". It is very opinionated shim layer between our code repositories and the tools we use to build and push the Dockerfile, generate the Kubernetes YAML, and get that YAML applied to the Kubernetes cluster. KTool can be pulled into the pipeline, as a gem or a docker image, and be called as an executable by whatever CI tool elects to bring it in. This gives us the added ability to change and add more features to KTool without having to chase around various GitHub Actions, CircleCI orbs, or other tooling-specific components.
The above diagram is a bit of a simplification; depending on the service and the workflow, there's any number of various test/validate/wait steps that can fit in at any time. There's also a separate process for deploying feature branches to atomic dev environments, but that warrants its own blog post.

Dockerfile

The Dockerfile lives in the Git repository, so developers can add dependencies as needed. We do single-stage builds; that is, we build the Dockerfile once, and reuse the image as we promote it from dev to staging and prod. That way, we're certain that the artifacts going to prod are the same as the artifacts that we tested in staging. We Docker images in an Amazon Elastic Container Registry accessible to the dev, staging, and production Kubernetes clusters. As a best practice, we avoid use of the :latest tag, and instead tag our images with the commit hash of the repository, so we know exactly which image corresponds to which PR. Our repositories are set to immutable, so we can't accidentally overwrite a production image with something else.
From the CI pipeline's point of view, all they're doing is running "KTool build".

Kubernetes Manifest

We don't expect developers to write their entire Kubernetes manifests; even if we did, there would have to be some automation to put the Docker image tag (which, as mentioned above, is a commit hash) into the pod specs. Instead, we ask developers to populate a simple YAML file that answers platform-agnostic questions like "does my webapp have an ingress?", "do I have a background worker too?", "how much RAM does it have?", "what command should it run?", and "what secrets should I bring in?". KTool then picks and choses some .yaml.erb templates we maintain and populates them with the right values. It then feeds them into Kustomize to collate them all together, insert the image tag and application name, and make a nice, compliant YAML file ready to go into a Kubernetes cluster. It also creates an ArgoCD application definition; more on that later.
This gives us the added advantage of being able to update KTool with new "sane defaults"; say, one day we decide to turn on read-only filesystems, or mandate that everything have a reverse proxy sidecar to do service mesh-y things. We update KTool, and everyone just gets that in their manifest next time they deploy.

Deployment to Kubernetes

Once we have the manifest, KTool checks it into yet another Git repository, which holds all of our Kubernetes manifests.
It is important to us that we version-control every manifest that goes into Kubernetes, so we know when an update happened, and how to roll back. It also helps identify config drift if someone did something by hand.
Speaking of config drift: we use ArgoCD as our means of actually getting our manifests applied to the cluster. It is pointed at the aforementioned repository full of YAML manifests, and as soon as something changes in the repository, it makes it so in Kubernetes. Not only does this mean that changes are automatically applied, but it reverts config drift, heals resources that were deleted by mistake, and provides a friendly GUI for developers looking to see how their services are behaving.
This way, even if we somehow accidentally lose an entire EKS cluster, we can be confident that everything will come back if we install ArgoCD and point it at the repository.

Impact

The unlocks from our move to Kubernetes are hard to count. Some of these deserve their own blog posts, but here's a few quick benefits in a nutshell.

Developers can update their runtime dependencies. Moving to a new version of Ruby or Python is as easy as updating the Dockerfile. No more need to get a server built or modified.
Developers can install dependencies for their runtime. Need a specific library or binary? Go in the Dockerfile and install it. No more need to ask someone to update a Chef cookbook.
Since pods just grow right back if they're disrupted, and we don't store any state on the hard drive, there's absolutely no reason why we can't use spot instances... so, our entire dev and staging environments are running on spot requests!
If it's time to update an AMI (which, in this case, really means the AWS-managed EKS worker AMIs), we modify our managed worker groups, and the change is rolled out automatically; pods are rescheduled from old nodes to new nodes, and because we use PodDisruptionBudgets in our Deployment resources, it happens without downtime. Again, because there's nothing persistent on the compute instances other than the Docker images themselves, moving the pods around is trivial.
We can scale horizontally at the push of a button (by updating the Deployment spec) or automatically (using Horizontal Pod Autoscaler). Again, this blog post only touches on the surface of our Kubernetes iceberg; there's a tremendous number of little implementation details, process improvements, and discoveries that came as part of this transition. Many of them do deserve their own blog post, and now that we've set some context, those entries can come. Stay tuned!

Trigger CircleCI Workflow. AKA Simple Deploy Button

Ken Collins — Sun, 05 Feb 2023 14:29:28 +0000

Very simple, no parameters needed, no enums, no booleans... just a really easy way to trigger a deploy with CircleCI. We can do this making use of the trigger_source pipeline value. When you click the button in CircleCI to "Trigger Pipeline" the value would be api vs something like webhook.

version: 2.1
jobs:
  deploy:
    machine:
      image: ubuntu-2204:current
    steps:
      - run: echo 'Deploying...'
workflows:
  deploy:
    when: { equal: [ api, << pipeline.trigger_source >> ] }
    jobs:
      - deploy

If your workflow needs a test job, consider doing something a bit more complicated. Here we use two when conditions to work with a parameter.

version: 2.1
parameters:
  workflow:
    type: enum
    default: test
    description: The workflow to trigger.
    enum: [test, deploy]
jobs:
  test-job:
    machine:
      image: ubuntu-2204:current
    steps:
      - run: echo 'Testing...'  
  deploy-job:
    machine:
      image: ubuntu-2204:current
    steps:
      - run: echo 'Deploying...'
workflows:
  test:
    when: { equal: [ test, << pipeline.parameters.workflow >> ] }
    jobs:
      - test-job
  deploy:
    when: { equal: [ deploy, << pipeline.parameters.workflow >> ] }
    jobs:
      - deploy-job

Now your CircleCI config will run tests by default and you can easily trigger a deploy via any branch using the "Trigger Pipeline" button.

New Amazon Linux Dev Container Features

Ken Collins — Mon, 31 Oct 2022 01:48:24 +0000

🆕 Want to use Codespaces with Amazon Linux 2? Check out customink/codespaces-features for two custom features. 1) common-amzn 2) docker-in-docker-amzn.

So, last year I shared how we could integrate Codespaces into our AWS Lambda docker compose patterns. Since then Microsoft's Development Containers specification has come a LONG way. Everything is wrapped up nice and neatly at the containers.dev site. Take a look if you have not already seen it.

Dev Containers?

So why are Development Containers & Codespaces such a big deal? I can illustrate some Lambda & Kubernetes use cases below, but first I would like to spell out a few features that may be new to some. Including existing Codespaces users.

The Dev Container specification at the lowest level of Codespaces is open to everyone and lots of tooling exists around it by a growing community.
The specification has a reference implementation via a node Command Line Interface. Think of this devcontainer CLI as a higher order docker compose. You can use this to run Codespaces projects locally!
Atop of the CLI, there is CI tooling for GitHub Actions. This means you can use the same development container as your CI/CD environment.

Containers Usage Areas

So where are containers used in your organization or projects? Here are some phases that most of us can identify with. Where projects move from the left to the right.

Development: Most of us have tried to use docker or compose at some point. For example, the most common use of this area would be running a database like MySQL. Docker makes this easy.

CI/CD: Typically where we run tests and hopefully build and/or deploy our code to production. If you have used CircleCI before, again a database container here might feel familiar.

Runtime: Which is often our final container environment. We can think of this as production for most of us but it could be any container orchestration like Kubernetes, Lambda, or Fargate.

Old AWS SAM Patterns with Docker Compose

Today our Lambda SAM cookiecutters leverage SAM's build image via docker compose to ensure local development is within the same environment for our CI/CD tooling. We ended up with something like this image

At the bottom we can see the host platform typically associated with each of these stages. Because we use Docker, we can be cross-platform and consistent. The problem? ⚠️ Making up your own docker/compose patterns are a huge drag. From SSH patterns to Docker-in-Docker gotchas.

New AWS SAM Patterns with Dev Containers

In the coming weeks the Lamby Cookiecutter will be updated to use Development Containers so folks with (or without) Codespaces can easily work with the project. The result will be something like this.

With Development Containers, so much docker compose boilerplate can be removed. Thanks in huge part to our newly released common & docker-in-docker Amazon Linux 2 features work. If you want to see an example of how this helps everyone including running Codespaces locally with VS Code, checkout our Crypteia Project's Development section. You can even use all this without VS Code & GitHub Codespaces. For example:

devcontainer build --workspace-folder .
devcontainer up --workspace-folder .
devcontainer run-user-commands --workspace-folder .
devcontainer exec --workspace-folder . ./bin/setup
devcontainer exec --workspace-folder . ./bin/test-local

Unexplored Development Container Space

So can Development Containers be used in your projects without the Lambda patterns above? Yes! Consider the following diagram that has a Platform Engineering team building base images.

These teams typically approach containers from the right to the left. Where base OS images are made into language specific images with variants for CI/CD. Just like SAM has build and runtime images. Technically for them "Runtime" is some container registry like Amazon ECR, but you get the idea.

At Custom Ink we are using our CircleCI images for our Kubernetes projects with Codespaces. The Microsoft team makes this easy since all of their features work with Ubuntu out of the box.

If your development stages look something like the image above. Please consider adopting Development Containers based on your CI/CD images and explore that big purple space by connecting your container value chain in a beautiful little circle. Thanks for reading!

Lambda Rust Extension for any Runtime to preload SSM Parameters as Secure Environment Variables!

Ken Collins — Wed, 29 Jun 2022 12:24:04 +0000

ℹ️ Crypteia Hits v1.0.0 Miletstone! 🎉 - It now has support for Python among other popular languages like Ruby, Node, & PHP. Crypteia is easy to install as a Lambda Layer or in a Container. It can even be used with K8s containers!

Crypteia is a new super fast Lambda Extension written in Rust which turns your serverless environment variables from SSM Parameter Store paths like these...

Environment:
  Variables:
    SECRET: x-crypteia-ssm:/myapp/SECRET

... into real environment variables when using your Runtime's language of choice. For example, assuming the SSM Parameter path above returns 1A2B3C4D5E6F as the value. Your code's environment variable methods would return that same value.

process.env.SECRET   // 1A2B3C4D5E6F
ENV['SECRET']        # 1A2B3C4D5E6F

It works by using a shared object library via the LD_PRELOAD environment variable in coordination with our Lambda Extension binary that loads all Parameter Store values within a few milliseconds of your function starting up.

Installation

When building your own Lambda Containers, download both the crypteia binary and libcrypteia.so shared object files that match your platform from our Releases page. Target platforms include the following using these naming conventions.

Amazon Linux 2: crypteia-amzn.zip & libcrypteia-amzn.zip
Debian, Ubuntu, Etc: crypteia-debian.zip & libcrypteia-debian.zip

⚠️ When building your own Lambda Containers, please make sure glibc is installed since this is used by redhook.

⚠️ For now our project supports the x86_64 architecture, but we plan to release arm64 variants soon. Follow or contribute in our GitHub Issue which tracks this topic.

Once these files are downloaded, they can be incorporated into your Dockerfile file like so:

RUN mkdir -p /opt/lib
RUN mkdir -p /opt/extensions
COPY crypteia /opt/extensions/crypteia
COPY libcrypteia.so /opt/lib/libcrypteia.so
ENV LD_PRELOAD=/opt/lib/libcrypteia.so

Lambda Layer

Our Amazon Linux 2 files can be used within a Lambda Layer that you can deploy to your own AWS account. You can use this project to build, publish, and deploy that layer since it has the SAM CLI installed. All you need to do is supply your own S3 bucket. For example:

aws configure
./amzn/setup
S3_BUCKET_NAME=my-bucket ./layer/deploy

Usage

First, you will need your secret environment variables setup in AWS Systems Manager Parameter Store. These can be whatever hierarchy you choose. Parameters can be any string type. However, we recommend using SecureString to ensure your secrets are encrypted within AWS. For example, let's assume the following paramter paths and values exists.

/myapp/SECRET -> 1A2B3C4D5E6F
/myapp/access-key -> G7H8I9J0K1L2
/myapp/envs/DB_URL -> mysql2://u:p@host:3306
/myapp/envs/NR_KEY -> z6y5x4w3v2u1

Crypteia supports two methods to fetch SSM parameters:

x-crypteia-ssm: - Single path for a single environment variable.
x-crypteia-ssm-path: - Path prefix to fetch many environment variables.

Using whatever serverless framework you prefer, setup your function's environment variables using either of the two SSM interfaces from above. For example, here is a environment variables section for an AWS SAM template that demonstrates all of Crypteia's features.

Environment:
  Variables:
    SECRET: x-crypteia-ssm:/myapp/SECRET
    ACCESS_KEY: x-crypteia-ssm:/myapp/access-key
    X_CRYPTEIA_SSM: x-crypteia-ssm-path:/myapp/envs
    DB_URL: x-crypteia
    NR_KEY: x-crypteia

When your function initializes, each of the four environmet variables (SECRET, ACCESS_KEY, DB_URL, and NR_KEY) will return values from their respective SSM paths.

process.env.SECRET;       // 1A2B3C4D5E6F
process.env.ACCESS_KEY;   // G7H8I9J0K1L2
process.env.DB_URL;       // mysql2://u:p@host:3306
process.env.NR_KEY;       // z6y5x4w3v2u1

Here are a few details about the internal implementation on how Crypteia works:

When accessing a single parameter path via x-crypteia-ssm: the environment variable name available to your runtime is used as is. No part of the parameter path effects the resulting name.
When using x-crypteia-ssm-path: the environment variable name can be anything and the value is left unchanged.
The parameter path hierarchy passed with x-crypteia-ssm-path: must be one level deep and end with valid environment variable names. These names must match environement placeholders using x-crypteia values.

For security, the usage of DB_URL: x-crypteia placeholders ensures that your application's configuration is in full control on which dynamic values can be used with x-crypteia-ssm-path:.

Lambda Layer

Shown below is a simple Node.js 16 function which has the appropriate IAM Permissions and Crypteia Lambda Layer added. Also configured are the needed LD_PRELOAD and SECRET environment variables. The code of this function log the value of the process.env.SECRET which does correctly resolve to the value within SSM Parameter Store.

Thank You 💞

Let me know if you find Crypteia useful or have any questions.

Getting Started with GitHub Codespaces from a Serverless Perspective

Ken Collins — Sun, 06 Feb 2022 22:37:15 +0000

If you are into Serverless and AWS Lambda, you may already know that the AWS Serverless Application Model (SAM) CLI makes it easy to leverage their Docker build images as development containers. We do exactly this for our Rails & Lambda projects.

Leveraging Docker with SAM ensures we have a Linux environment and versioned dependencies that closely mimic the Lambda Runtime or Container being shipped. The use and The Promise of Docker to solve these problems is nothing new... but something else is.

✨ The Rise of Ephemeral Dev Environments

A few weeks ago GitHub's engineering team released an in-depth article announcing their internal usage of the now generally available GitHub Codespaces. Since Custom Ink shares many of the same problems described in this post, I was curious if our Lambda projects could easily leverage Codespaces. But what is this new tool? Where did it come from? And what is this devcontainer.json file?

As best I can tell this all started in May of 2019 when the VS Code team first mentioned their remote development extensions. About a year later this content was rolled up into the VS Code Remote Development guides we have today.

Prior to Codespaces, we have had a clear leader in the automated development environment space with Gitpod. It was even featured in a January 2021 episode of Containers from the Couch. Gitpod leverages the same technology built into VS Code for remote development.

However, sometimes slow and steady wins the race. If this were ever true for GitHub-based projects, I think we have a huge winner with GitHub Codespaces. Keep reading below on how your company (or you) could get started. I will even cover how well Codespaces has worked for our Lambda projects that use an existing Docker in Docker development pattern.

⚙️ GitHub Settings

GitHub Codespaces is ONLY available now for GitHub Teams & Enterprise Cloud plans. It is not yet available for public repositories. If you are an administrator of such an account, here are a few things I did at the organization level to get started experimenting.

Enable Codespaces: This can also be disabled completely or enabled for select users.
Repository Access: You can even limit repositories that are able to use Codespaces. If your GitHub account leverages permissions & teams, remember, Codespaces (via the generated GITHUB_TOKEN will not grant anyone elevated permissions to other repositories.
Manage Spending Limits: It would have been neat to see a way to limit which VMs (vCPU/Memory) options could have been used here.
Organizational Secrets: Create any secrets your organization needs to enable individuals to work. Remember, Codespaces secrets can be set at the repository or even user level too. Pick the one(s) that work the best for y'all.

🔰 Developer Tips

It could go without saying but getting good at Codespaces for most may mean getting good at VS Code. Technically you could bring your own editor like Vim or Emacs. But trust me, as a recent Sublime Text convert, switching to VS Code is worth it. Make sure to take the time to Google, learn, and in some cases install packages that make the transition easier.

Dotfiles & Settings

Remote development needs to feel local! Everything that makes your editor & terminal productive needs to be available to you. As described in the Personalizing Codespaces guide setting up your Dotfiles was high on my list.

For years I have maintained a personal Zshkit which had a ton of personal functions and aliases. When moving to Codespaces, I took the time to clean them up and create a github.com/metaskills/dotfiles repository, cloned it locally and hooked it up to my ZSH (default shell on Mac) ~/.zshrc file. Codespaces will automatically clone this repo when creating a Codespace and install it by running the install.sh script. Example.

if [ "$CODESPACES" = "true" ]; then
  echo "source /workspaces/.codespaces/.persistedshare/dotfiles/rc" >> $HOME/.zshrc
  sudo chsh -s /usr/bin/zsh
fi

You can leverage the CODESPACES environment variable to do any customization per environment. Also, do not forget to use Settings Sync. I think this is only needed if you use VS Code's web-based editor. More on that topic later.

Your Codespaces Settings

You can Manage Your Codespaces settings at somewhat the same level as the organization. Here are a few settings I did.

Access & Security: Set this to "All repositories". Your needs may vary.
Editor Preference: Set to "Visual Studio Code" vs for web. Ensures the [<> Code] button on repos opens VS Code on my Mac and avoids the need to click redirect in the browser.
Region: I set this manually to EastUs but I suspect I had no reason to do so.
Added Secrets: Read below on using SSH with Ruby Bundler or NPM packages.

Codespaces Extension

Install the GitHub Codespaces for VS Code. I think this is done for you automatically if you are using the web-based editor. Installing it on your host machine's VS Code will mean you can use Codespaces without ever browsing to GitHub.com and clicking on a [<> Code] button.

The Integrated Terminal

Assuming you have setup your Dotfiles, VS Code's integrated terminal should feel familiar by mirroring your host machine's prompt, aliases, and more. If your default shell is ZSH, you may need to do a few things to help Codespaces use ZSH by default vs Bash. Here are my settings for the integrated terminal now. Mind you, there was (maybe still is) a bug in VS Code where ZSH would not be respected. I have noticed in some cases Bash is used but it is easy to launch a new profile with ZSH if that happens.

  "terminal.integrated.fontSize": 14,
  "terminal.integrated.defaultProfile.osx": "zsh",
  "terminal.integrated.defaultProfile.linux": "zsh",

Using Command+K to clear the terminal's buffer is second nature to most. By default this key binding will not reach the integrated terminal. You can edit your Keyboard Shortcuts JSON file to solve for that. Below is a screen capture of the magic little button you have to press to edit that raw JSON file. Use the following snippet to fix this.

{
  "key": "cmd+k",
  "command": "workbench.action.terminal.clear",
  "when": "terminalFocus"
}

Terminal visibility and placement. When working on my laptop's smaller screen, I learned that you can use Control+~ to toggle the visibility of the integrated terminal. However, when working at my desk and larger screen, I really want the integrated terminal to be to the right of my editor. Thanks to this this Stack Overflow here are convoluted steps to make this happen. Hopefully one day they will make this easier. 😅

At the right top of the integrated terminal, click the + sign to open a 2nd terminal.
Within the panel to the right, right click any of the two profiles, select Move into Editor Area.
Close the bottom integrated terminal with the x button.
Focus the editor tab at the top moved from step 2, click the [|] split editor button.
Close the shell tab on the left side of the screen.

🎉 Fun Highlights

Here are a few things I was pleasantly surprised with Codespaces' DX and how it works:

When learning Codespaces or working on uncommitted code, you may have to rebuild your development container. Codespaces automatically maintains your present working directory, open files, etc when doing this. Amazing!
You can see all your Codespaces on GitHub by navigating to https://github.com/codespaces. However, I typically use VS Code's extension to navigate, open, and disconnect.
Leveraging the CODESPACES environment variable set to true is an easy way to integrate your existing tooling into Codespaces allowing your teams to support multiple ways to bootstrap your applications.
Forwarded ports are automatically detected via the integrated terminal's STDOUT. For example, a .bin/rails server will ouput whatever host/port you are using and Codespaces will see it. If needed you can use the forwardPorts config for devcontainer.json.

⚠️ Difficult Lessons

Some hard lessons learned when dipping into the deep end of using GitHub Codespaces. If you have any to share, please drop some comments below.

Private Packages & SSH

GitHub does a great job at providing your Codespace with a short lived GITHUB_TOKEN. Most package managers including NPM and Bundler can leverage this. However, if your organization has standardized on SSH setting up your projects could be a problem.

Thankfully when I reached out on Twitter, Jonathan Carter on the Codespaces team, seemed to suggest they may be working on a native SSH integration one day. Till then, here is the solution I came up with. This process address some sequencing issues around devcontainer.json's Lifecycle Scripts and when your Dotfiles are installed. Credit to VS Codes Using SSH Keys guide. Also, some things here are pulled directly from the GitHub Action to setup SSH. Again, thanks to Johnathan Carter for the ideas.

Create a personal Codespace secret called PERSONAL_SSH_KEY by visiting this page https://github.com/settings/codespaces/secrets/new and adding your private key, typically found in the ~/.ssh/id_rsa file.
Add this snippet to your postCreate script. It ensures GitHub is in the known hosts for SSH.

echo "Adding GitHub.com keys to ~/.ssh/known_hosts"
printf "\ngithub.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==\n" >> ~/.ssh/known_hosts
printf "\ngithub.com ssh-dss AAAAB3NzaC1kc3MAAACBANGFW2P9xlGU3zWrymJgI/lKo//ZW2WfVtmbsUZJ5uyKArtlQOT2+WRhcg4979aFxgKdcsqAYW3/LS1T2km3jYW/vr4Uzn+dXWODVk5VlUiZ1HFOHf6s6ITcZvjvdbp6ZbpM+DuJT7Bw+h5Fx8Qt8I16oCZYmAPJRtu46o9C2zk1AAAAFQC4gdFGcSbp5Gr0Wd5Ay/jtcldMewAAAIATTgn4sY4Nem/FQE+XJlyUQptPWMem5fwOcWtSXiTKaaN0lkk2p2snz+EJvAGXGq9dTSWHyLJSM2W6ZdQDqWJ1k+cL8CARAqL+UMwF84CR0m3hj+wtVGD/J4G5kW2DBAf4/bqzP4469lT+dF2FRQ2L9JKXrCWcnhMtJUvua8dvnwAAAIB6C4nQfAA7x8oLta6tT+oCk2WQcydNsyugE8vLrHlogoWEicla6cWPk7oXSspbzUcfkjN3Qa6e74PhRkc7JdSdAlFzU3m7LMkXo1MHgkqNX8glxWNVqBSc0YRdbFdTkL0C6gtpklilhvuHQCdbgB3LBAikcRkDp+FCVkUgPC/7Rw==\n" >> ~/.ssh/known_hosts

Add this snippet to your Dotfiles. It will ensure the proper SSH agent is started, if not already, and that the key environment variables are set.

if [ "$CODESPACES" = "true" ]; then
  if [ -z "$SSH_AUTH_SOCK" ]; then
    RUNNING_AGENT="`ps -ax | grep 'ssh-agent -s' | grep -v grep | wc -l | tr -d '[:space:]'`"
    if [ "$RUNNING_AGENT" = "0" ]; then
      # Launch a new instance of the agent
      ssh-agent -s &> $HOME/.ssh/ssh-agent
    fi
    eval `cat $HOME/.ssh/ssh-agent`
  fi
  # Add my SSH key.
  if [ -n "${PERSONAL_SSH_KEY+1}" ]; then
    ssh-add - <<< "${PERSONAL_SSH_KEY}"
  fi
fi

In order to see this all come together with our Docker in Docker Lambda patterns, please read the Serverless Docker Patterns article in this series where we describe how to use the SSH_AUTH_SOCK in a cross platform way for Mac & Linux.

AWS CLI

For our Lambda projects we use Docker in Docker patterns where both the AWS & SAM CLIs are pre-installed on the development image. However, you may need the AWS CLI installed on the developer's host machine too. In this case, Codespaces. Here is a short snippet that you can use in your postCreate script.

echo "Installing AWS CLI"
pushd /tmp
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip -qq awscliv2.zip
sudo ./aws/install
rm -rf awscliv2.zip ./aws
popd

Docker in Docker

I've said this before but cross platform Docker in Docker is really hard. This series aims to talk about most of them, but one I learned the hard way is that sometimes the pain comes from the ones you love... in this case AWS SAM. The team is doing some amazing work but I ran into a few issues where Docker in Docker patterns have broken down. Read here for details.

🚂 Full Lamby Example

Assuming the other patterns were in place like various postCreate hooks for SSH, using GitHub Codespaces with your already Docker'ized project is super easy. Here is a complete .devcontainer/devcontainer.json file for one of our projects. Again, see the Serverless Docker Patterns related post on how we are using COMPOSE_FILE for Mac filesystem performance and why it would be needed here.

{
  "name": "my-application",
  "forwardPorts": [4020],
  "remoteEnv": {
    "COMPOSE_FILE": "docker-compose.yml"
  },
  "postCreateCommand": "./.devcontainer/postCreate"
}

In fact, none of this would be needed for a starter application! Give it a try. Go through our Lamby Quick Start guide, commit your project to GitHub... and give Codespaces a try!

🔐 Security Questions

The Codespaces team was kind enough to write their own Security in Codespaces documentation. I'll highlight their introduction below:

Codespaces is designed to be security hardened by default. Consequently, you will need to ensure that your software development practices do not risk reducing the security posture of your codespace.

This guide describes the way Codespaces keeps your development environment secure and provides some of the good practices that will help maintain your security as you work. As with any development tool, remember that you should only open and work within repositories you know and trust.

Good stuff! Security is a shared responsibility and it appears GitHub is doing their part. Please read over the full documentation for more information, but here are a few things I paid special attention to.

Audit Logs: Are generated and can be queried.
Organization & User Secrets: Built on the same technology GitHub draws a line between GitHub standard org/user secrets with the Codespace ones. Again, they can be set at the organization, repository, or user. Providing an immense amount of control and security layers.
Dotfiles: Remind users that these are public repositories! Tho possible to encrypt secrets, I personally recommend keeping them basic to aliases and functions.
Secure Networking: Authenticated via GitHub via temporary tokens. Forwarding ports for web servers is done securely over the network between the host. Nothing is public by default.

🔮 What is Coming?

As mentioned above, I would love to see a native SSH solution. For now, the workarounds are minimal and feel secure with GitHub Secrets and Codespaces integration.

In their introductory blog article, the GitHub team put a lot of emphasis on prebuilds ensuring that each Codespaces development environment was super fast to setup. This was critical for their team and as of now Gitpod is making a clear distinction this is a key differentiator for them. I suspect prebuilds are coming soon. 🤔

📚 Resources

Thanks so much for reading! I would love to hear if you found this article helpful or what your organization may be doing with GitHub Codespaces. 💕

GitHub Codespaces - Blazing fast cloud developer environments with Visual Studio Code backed by high performance VMs that start in seconds.
GitHub’s Engineering Team has moved to Codespaces - Great description of the business needs for easy development environments. Common for most orgs.
Getting Started with Rails on Lambda - An quick start guide using Docker for development with GitHub & Codespaces.
VS Code Remote Development - The architecture behind GitHub Codespaces.
Gitpod - Spin up fresh, automated dev environments for each task, in the cloud, in seconds.
DevX Digest: The Rise of Ephemeral Developer Environments - Great post by Pauline P. Narvas on where cloud-based dev environments are headed.

AWS Systems Manager (SSM) Cross Region Replication

Katherine (she/her) — Wed, 12 Jan 2022 20:07:20 +0000

Overview of SSM Replication

This blog post will explain in detail how to set up cross region replication for AWS Parameter Store. As of the writing of this blog post, AWS does not have a native feature for replicating parameters in SSM. If you are using SSM Parameter Store instead of Secrets Manager and are seeking a way to replicate parameters for DR/Multi-Region purposes, this post may help you.

Diagram showing the architecture setup:

Serverless Framework Setup

I used Lamby cookie-cutter as the framework for this Lambda, which made a lot of the initial set up very easy! Please take a look at that site & set up your serverless framework for the work to be done ahead. I will first share the CloudFormation template used, then share the code that makes the replication work as well as plain in detail what's happening.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: AWS SSM regional replication for multi-region setup
Parameters:
  StageEnv:
    Type: String
    Default: dev
    AllowedValues:
      - test
      - dev
      - staging
      - prod
Mappings:
  KmsMap:
    us-east-1:
      dev: 'arn:aws:kms:us-east-1:123456:key/super-cool-key1'
      staging: 'arn:aws:kms:us-east-1:123456:key/super-cool-key2' 
      prod: arn:aws:kms:us-east-1:123456:key/super-cool-key3'
    us-east-2:
      dev: 'arn:aws:kms:us-east-2:123456:key/super-cool-key1'
      staging: 'arn:aws:kms:us-east-2:123456:key/super-cool-key1'
      prod: 'arn:aws:kms:us-east-2:123456:key/super-cool-key1' 
  DestinationMap:
    us-east-1: 
      target: "us-east-2"
Resources:
  ReplicationQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub 'SSM-SQS-replication-${StageEnv}-${AWS::Region}'
      VisibilityTimeout: 1000
  LambdaRegionalReplication:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: lib/ssm_regional_replication.handler
      Runtime: ruby2.7
      Timeout: 900
      MemorySize: 512
      Environment:
        Variables:
          STAGE_ENV: !Ref StageEnv
          TARGET_REGION: !FindInMap [DestinationMap, !Ref AWS::Region, target]
          SKIP_SYNC: 'skip_sync'
      Events:
        InvokeFromSQS:
          Type: SQS
          Properties:
            Queue: {"Fn::GetAtt" : [ "ReplicationQueue", "Arn" ]}
            BatchSize: 1
            Enabled: true
        ReactToSSM:
          Type: EventBridgeRule
          Properties:
            Pattern:
              detail-type:
                - Parameter Store Change 
              source:
                - aws.ssm
      Policies:
      - Statement:
        - Sid: ReadSSM
          Effect: Allow
          Action:
          - ssm:GetParameter
          - ssm:GetParameters
          - ssm:PutParameter
          - ssm:DeleteParameter
          - ssm:AddTagsToResource
          - ssm:ListTagsForResource
          Resource: 
          - !Sub "arn:aws:ssm:*:${AWS::AccountId}:parameter/*"
      - Statement:
        - Sid: DecryptSSM
          Effect: Allow
          Action:
          - kms:Decrypt
          - kms:Encrypt
          Resource: 
          - !FindInMap [KmsMap, us-east-1, !Ref StageEnv]
          - !FindInMap [KmsMap, us-east-2, !Ref StageEnv]
  LambdaFullReplication:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: lib/ssm_full_replication.handler
      Runtime: ruby2.7
      Timeout: 900
      MemorySize: 512
      Environment:
        Variables:
          STAGE_ENV: !Ref StageEnv
          TARGET_REGION: !FindInMap [DestinationMap, !Ref AWS::Region, target]
          SKIP_SYNC: 'skip_sync'
      Events:
        DailyReplication:
          Type: Schedule
          Properties:
            Description: Cronjob to run replication at 9:30am EST every Wednesday (cron is UTC)
            Enabled: True 
            Name: DailySSMReplication
            Schedule: "cron(30 13 ? * 4 *)"
      Policies:
      - Statement:
        - Sid: SQSPerms
          Effect: Allow
          Action:
          - sqs:SendMessage
          Resource: 
          - !Sub "arn:aws:sqs:*:${AWS::AccountId}:SSM-SQS-replication-*"
      - Statement:
        - Sid: ReadSSM
          Effect: Allow
          Action:
          - ssm:GetParameter
          - ssm:GetParameters
          - ssm:PutParameter
          - ssm:AddTagsToResource
          - ssm:ListTagsForResource
          - ssm:DescribeParameters
          Resource: 
          - !Sub "arn:aws:ssm:*:${AWS::AccountId}:*"
          - !Sub "arn:aws:ssm:*:${AWS::AccountId}:parameter/*"
      - Statement:
        - Sid: DecryptSSM
          Effect: Allow
          Action:
          - kms:Decrypt
          - kms:Encrypt
          Resource: 
          - !FindInMap [KmsMap, us-east-1, !Ref StageEnv]
          - !FindInMap [KmsMap, us-east-2, !Ref StageEnv]

The above template does a number of things. It creates my SQS queue, a regional replication lambda that is event based, and a full replication lambda that is cron based. Under the 'Mappings' section I have "KmsMap" which maps to the aws/ssm KMS keys. If you use other keys for your SSM entries, enter that value here. If you use many keys across your SSM parameters, simply add them to the lambda properties, example here:

      - Statement:
        - Sid: DecryptSSM
          Effect: Allow
          Action:
          - kms:Decrypt
          - kms:Encrypt
          Resource: 
          - !FindInMap [KmsMap, us-east-1, !Ref StageEnv]
          - !FindInMap [KmsMap, us-east-2, !Ref StageEnv]
          - 'arn:aws:kms:us-east-1:123456:key/my-managed-key1'

The other 'Mapping', DestinationMap, sets up my source and target region. My original SSM parameters are in us-east-1, so the target is us-east-2 in this case. The SQS queue holds all of the parameters from the LambdaFullReplication, since lambdas cannot run indefinitely, there's a high chance the function won't finish before going through all of your parameters. This LambdaFullReplication function sends the parameters to the SQS queue, where the LambdaRegionalReplication then performs the put action to the destination region. The VisibilityTimeout is set to 1000 to allow some wiggle room for the lambda (900).
The full replication lambda runs every Wednesday (or whatever frequency you'd like) for a few reasons:

to do the initial get/put for the parameters and
to catch any parameters that have/delete the skip_sync tag

I will discuss the skip_sync tag in detail when discussing the code. The regional replication lambda runs when there's an entry in the SQS queue that has to be processed, or anytime there's a change to a parameter, driven by event based actions.

Code Setup

Next I will discuss and share the Ruby code that actually does the work. There are three Ruby files that make this lambda function, parameter_store.rb, ssm_regional_replication.rb, and ssm_full_replication.rb. I will share the code along with the comments around what is happening in the file.

require 'aws-sdk-ssm'
# Create ParameterStore class, to be shared by both regional
# and full replication lambda.
class ParameterStore
  # The parameter store class creates instance variables with "attr_accessor" 
  # for the initial client, response, name, and tag_list. 
  attr_accessor :client, :response, :name, :tag_list
  # Initialize method for hash
  # this allows the client & name instance vars
  # to be used outside of the init method
  def initialize(h)
    self.client = h[:client] # this gets the client key from CloudWatch metrics
    self.name = h[:name] # gets the name of the param & assigns it to name instance var
  end
  # this method takes the client & name args from prev method.
  def self.find_by_name(client, name) 
    # create new client connection & name from private `find_by_name` method
    new(client: client, name: name).find_by_name(name)
  end
  private 
  def find_by_name(name)
    # set begin block in order for the get_parameter call to
    # loop through all of the parameters
    begin
      # declare instance variable with self.response
      # set to the AWS client connection calling
      # get_parameter method via Ruby CLI
      # extract the name & with_decruption options set
      self.response = client.get_parameter({
        name: name,
        with_decryption: true,
      })
      # rescue to look for AWS SSM throttling errors.
      # take the exception below, and place in variable "e"
    rescue Aws::SSM::Errors::ThrottlingException => e 
      p "Sleeping for 60 seconds while getting parameters."
      sleep(60)
      # will re-run what is in begin block
      retry
    end
    self
  end

  # creates a `tag_list` instance var
  # `||=` operator is Ruby "short-circuit" which means
  # if `tag_list` is set, then skip this part,
  # if not set, then set it to what is on the right side of equals sign.
  # the purpose is to set the tag_list var equal to
  # the response from the `list_tags_for_resource`¹ 
  # which contains resource_type set to Parameter, and the 
  # resource_id set to name
  def tag_list
    @tag_list ||= client.list_tags_for_resource({resource_type: 'Parameter', resource_id: name})
  end
  # checks the `tag_list` method above & runs a 
  # select method on the tag_list hash
  # loops to see if there is a key with the `key` value in hash
  # and checks presence of a `skip_sync` tag with the `.any?` 
  # boolean method. If this exists, then the lambda function
  # will not run and the replication will not occur.
  # If this does not exist, then it proceeds. 
  # You may want to skip syncing for regional specific resources. 
  # If you want to replicate an initial skip_sync param, simply
  # remove the tag in question and on the next run, the param will sync`
  def skip_sync?
    tag_list[:tag_list].select {|key| key[:key] == $skip_tag }.any?
  end
  # Calls the Ruby `put_parameters` method on the `client_target` parameter.
  # `put_parameter` replicates name, value, type, and overwrite. This method
  # also adds the tags copied over from the tag_list method to resources by name.
  def sync_to_region(client_target)
    client_target.put_parameter({
      name: response['parameter']['name'], # required
      value: response['parameter']['value'], # required
      type: response['parameter']['type'], # accepts String, StringList, SecureString
      overwrite: true,
    })
    client_target.add_tags_to_resource({resource_type: 'Parameter', resource_id: name, tags: tag_list.to_h[:tag_list]})    
  end
end

The next file I will discuss is the ssm_full_replication.rb piece of the code. As you may gather from the name, this is responsible for full replication.

# this pulls the AWS sdk gem
require 'aws-sdk-ssm'
require 'aws-sdk-sqs'
require_relative 'parameter_store'
# Declare global variables which are set to the
# respective values from CloudFormation template.
$target_region = ENV['TARGET_REGION'] or raise "Missing TARGET_REGION variable."
$skip_tag = ENV['SKIP_SYNC'] or raise "Missing skip_sync tag."
$stage_env = ENV['STAGE_ENV']
# method set to us-east-1 for source region. 
# var `sqs_client` set to new SQS client connection in target region
# var `sts_client` set to new STS client conn in source region.
# call `send_message` on `sqs_client` var with queue_url & message_body as params.
def send_params_to_sqs(name)
  region = "us-east-1"
  sqs_client = Aws::SQS::Client.new(region: $target_region)
  sts_client = Aws::STS::Client.new(region: region)
  sqs_client.send_message(
    queue_url: "https://sqs.#{region}.amazonaws.com/#{sts_client.get_caller_identity.account}/SSM-SQS-replication-#{$stage_env}-#{region}",
    message_body: name
  )
end
# sets new SSM client connection in source region
# and new SSM client_target connection in target region
def handler(event:, context:)
  client = Aws::SSM::Client.new
  client_target = Aws::SSM::Client.new(region: $target_region)
  # next_token set to nil, which is important at start of lambda func
  next_token = nil
  # loop starts with begin block which
  # runs before the rest of the code in method.
  loop do 
    begin
      # describe_batch is set to value from 
      # describe_parameters² call on the client variable.
      @describe_batch = client.describe_parameters({
        # parameter_filter limits request results to what we need
        parameter_filters: [
          {
            key: "Type",
            values: ["String", "StringList", "SecureString"]
          },
        ],
        # next_token is set to next set of items to return
        next_token: next_token,
      })
      # describe_batch var calls iterative loop and
      # sends param name to send_params_to_sqs method
      @describe_batch.parameters.each do |item|
        send_params_to_sqs(item.name)
      end
      # break means that func will end if the next_token value is empty.
      break if @describe_batch.next_token.nil?
      next_token = @describe_batch.next_token
      # exception handling. it looks for this error message, and this is how it will handle, by pausing for 60 seconds.
    rescue Aws::SSM::Errors::ThrottlingException
      p "Sleeping for 60 seconds while describing parameters."
      sleep(60)
    end
  end
end

The last file to share is the ssm_regional_replication.rb file. This file is event based and does the regional replication.

# this pulls the AWS sdk gem
require 'aws-sdk-ssm'
require_relative 'parameter_store'
# Global vars for file
$target_region = ENV['TARGET_REGION'] or raise "Missing TARGET_REGION variable."
$skip_tag = ENV['SKIP_SYNC'] or raise "Missing skip_sync tag."
# CloudWatch sends events in a specific format compared to SQS triggered lambdas
# so this method grabs the values from CloudWatch handles both formats.
def massage_event_data(event)
  # pull out values from a cloudwatch invocation
  operation = event.fetch('detail', {})['operation']
  name      = event.fetch('detail', {})['name']
  return operation,name if operation && name
  operation = 'Update' 
  name      = event.fetch('Records', []).first['body']
  return operation,name 
end
def handler(event:, context:)
  # set vars called operation and name. output from prev. method.
  # create new client & target vars for SSM
  operation,name = massage_event_data(event)
  client = Aws::SSM::Client.new
  client_target = Aws::SSM::Client.new(region: $target_region)
  # this logic runs event based code. If the operation from 
  # the CloudWatch metrics is equal to either update or create
  # the ps var uses the ParameterStore find_by_name class method
  # and passes the client * name.
  if operation == 'Update' || operation == 'Create'
    ps = ParameterStore.find_by_name(client, name)
    # if the ps var has a skip_sync tag, then the CloudWatch logs
    # you will get what's in the puts string. if there is no tag
    # it syncs to target region.
    if ps.skip_sync?
      puts "This function has been opted out, not replicating parameter."
    else
      ps.sync_to_region(client_target)
    end
  # if the operation is delete in the source region, then the delete_parameter method is called on the
  # client_target and it's also deleted from the target_region to ensure parity.
  elsif operation == 'Delete'
    response = client_target.delete_parameter({
      name: name, # required. go into event, reference the detail key, and the value name
    })
  end
end

References to AWS API docs page:

If you want to be sure that there are no missed variables, you can always set up a CloudWatch alarm on if your lambda has any failed invocations or if your SQS queue isn't sending any messages. I hope that this has helped others who are looking for a way to replicate SSM parameters in AWS from one region to another. That's the end of the code, I know it is a lot to digest, so if you have any questions please leave a comment and I'll do my best to follow up.