Forem: Katherine (she/her)

AWS Systems Manager (SSM) Cross Region Replication

Katherine (she/her) — Wed, 12 Jan 2022 20:07:20 +0000

Overview of SSM Replication

This blog post will explain in detail how to set up cross region replication for AWS Parameter Store. As of the writing of this blog post, AWS does not have a native feature for replicating parameters in SSM. If you are using SSM Parameter Store instead of Secrets Manager and are seeking a way to replicate parameters for DR/Multi-Region purposes, this post may help you.

Diagram showing the architecture setup:

Serverless Framework Setup

I used Lamby cookie-cutter as the framework for this Lambda, which made a lot of the initial set up very easy! Please take a look at that site & set up your serverless framework for the work to be done ahead. I will first share the CloudFormation template used, then share the code that makes the replication work as well as plain in detail what's happening.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: AWS SSM regional replication for multi-region setup
Parameters:
  StageEnv:
    Type: String
    Default: dev
    AllowedValues:
      - test
      - dev
      - staging
      - prod
Mappings:
  KmsMap:
    us-east-1:
      dev: 'arn:aws:kms:us-east-1:123456:key/super-cool-key1'
      staging: 'arn:aws:kms:us-east-1:123456:key/super-cool-key2' 
      prod: arn:aws:kms:us-east-1:123456:key/super-cool-key3'
    us-east-2:
      dev: 'arn:aws:kms:us-east-2:123456:key/super-cool-key1'
      staging: 'arn:aws:kms:us-east-2:123456:key/super-cool-key1'
      prod: 'arn:aws:kms:us-east-2:123456:key/super-cool-key1' 
  DestinationMap:
    us-east-1: 
      target: "us-east-2"
Resources:
  ReplicationQueue:
    Type: AWS::SQS::Queue
    Properties:
      QueueName: !Sub 'SSM-SQS-replication-${StageEnv}-${AWS::Region}'
      VisibilityTimeout: 1000
  LambdaRegionalReplication:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: lib/ssm_regional_replication.handler
      Runtime: ruby2.7
      Timeout: 900
      MemorySize: 512
      Environment:
        Variables:
          STAGE_ENV: !Ref StageEnv
          TARGET_REGION: !FindInMap [DestinationMap, !Ref AWS::Region, target]
          SKIP_SYNC: 'skip_sync'
      Events:
        InvokeFromSQS:
          Type: SQS
          Properties:
            Queue: {"Fn::GetAtt" : [ "ReplicationQueue", "Arn" ]}
            BatchSize: 1
            Enabled: true
        ReactToSSM:
          Type: EventBridgeRule
          Properties:
            Pattern:
              detail-type:
                - Parameter Store Change 
              source:
                - aws.ssm
      Policies:
      - Statement:
        - Sid: ReadSSM
          Effect: Allow
          Action:
          - ssm:GetParameter
          - ssm:GetParameters
          - ssm:PutParameter
          - ssm:DeleteParameter
          - ssm:AddTagsToResource
          - ssm:ListTagsForResource
          Resource: 
          - !Sub "arn:aws:ssm:*:${AWS::AccountId}:parameter/*"
      - Statement:
        - Sid: DecryptSSM
          Effect: Allow
          Action:
          - kms:Decrypt
          - kms:Encrypt
          Resource: 
          - !FindInMap [KmsMap, us-east-1, !Ref StageEnv]
          - !FindInMap [KmsMap, us-east-2, !Ref StageEnv]
  LambdaFullReplication:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: lib/ssm_full_replication.handler
      Runtime: ruby2.7
      Timeout: 900
      MemorySize: 512
      Environment:
        Variables:
          STAGE_ENV: !Ref StageEnv
          TARGET_REGION: !FindInMap [DestinationMap, !Ref AWS::Region, target]
          SKIP_SYNC: 'skip_sync'
      Events:
        DailyReplication:
          Type: Schedule
          Properties:
            Description: Cronjob to run replication at 9:30am EST every Wednesday (cron is UTC)
            Enabled: True 
            Name: DailySSMReplication
            Schedule: "cron(30 13 ? * 4 *)"
      Policies:
      - Statement:
        - Sid: SQSPerms
          Effect: Allow
          Action:
          - sqs:SendMessage
          Resource: 
          - !Sub "arn:aws:sqs:*:${AWS::AccountId}:SSM-SQS-replication-*"
      - Statement:
        - Sid: ReadSSM
          Effect: Allow
          Action:
          - ssm:GetParameter
          - ssm:GetParameters
          - ssm:PutParameter
          - ssm:AddTagsToResource
          - ssm:ListTagsForResource
          - ssm:DescribeParameters
          Resource: 
          - !Sub "arn:aws:ssm:*:${AWS::AccountId}:*"
          - !Sub "arn:aws:ssm:*:${AWS::AccountId}:parameter/*"
      - Statement:
        - Sid: DecryptSSM
          Effect: Allow
          Action:
          - kms:Decrypt
          - kms:Encrypt
          Resource: 
          - !FindInMap [KmsMap, us-east-1, !Ref StageEnv]
          - !FindInMap [KmsMap, us-east-2, !Ref StageEnv]

The above template does a number of things. It creates my SQS queue, a regional replication lambda that is event based, and a full replication lambda that is cron based. Under the 'Mappings' section I have "KmsMap" which maps to the aws/ssm KMS keys. If you use other keys for your SSM entries, enter that value here. If you use many keys across your SSM parameters, simply add them to the lambda properties, example here:

      - Statement:
        - Sid: DecryptSSM
          Effect: Allow
          Action:
          - kms:Decrypt
          - kms:Encrypt
          Resource: 
          - !FindInMap [KmsMap, us-east-1, !Ref StageEnv]
          - !FindInMap [KmsMap, us-east-2, !Ref StageEnv]
          - 'arn:aws:kms:us-east-1:123456:key/my-managed-key1'

The other 'Mapping', DestinationMap, sets up my source and target region. My original SSM parameters are in us-east-1, so the target is us-east-2 in this case. The SQS queue holds all of the parameters from the LambdaFullReplication, since lambdas cannot run indefinitely, there's a high chance the function won't finish before going through all of your parameters. This LambdaFullReplication function sends the parameters to the SQS queue, where the LambdaRegionalReplication then performs the put action to the destination region. The VisibilityTimeout is set to 1000 to allow some wiggle room for the lambda (900).
The full replication lambda runs every Wednesday (or whatever frequency you'd like) for a few reasons:

to do the initial get/put for the parameters and
to catch any parameters that have/delete the skip_sync tag

I will discuss the skip_sync tag in detail when discussing the code. The regional replication lambda runs when there's an entry in the SQS queue that has to be processed, or anytime there's a change to a parameter, driven by event based actions.

Code Setup

Next I will discuss and share the Ruby code that actually does the work. There are three Ruby files that make this lambda function, parameter_store.rb, ssm_regional_replication.rb, and ssm_full_replication.rb. I will share the code along with the comments around what is happening in the file.

require 'aws-sdk-ssm'
# Create ParameterStore class, to be shared by both regional
# and full replication lambda.
class ParameterStore
  # The parameter store class creates instance variables with "attr_accessor" 
  # for the initial client, response, name, and tag_list. 
  attr_accessor :client, :response, :name, :tag_list
  # Initialize method for hash
  # this allows the client & name instance vars
  # to be used outside of the init method
  def initialize(h)
    self.client = h[:client] # this gets the client key from CloudWatch metrics
    self.name = h[:name] # gets the name of the param & assigns it to name instance var
  end
  # this method takes the client & name args from prev method.
  def self.find_by_name(client, name) 
    # create new client connection & name from private `find_by_name` method
    new(client: client, name: name).find_by_name(name)
  end
  private 
  def find_by_name(name)
    # set begin block in order for the get_parameter call to
    # loop through all of the parameters
    begin
      # declare instance variable with self.response
      # set to the AWS client connection calling
      # get_parameter method via Ruby CLI
      # extract the name & with_decruption options set
      self.response = client.get_parameter({
        name: name,
        with_decryption: true,
      })
      # rescue to look for AWS SSM throttling errors.
      # take the exception below, and place in variable "e"
    rescue Aws::SSM::Errors::ThrottlingException => e 
      p "Sleeping for 60 seconds while getting parameters."
      sleep(60)
      # will re-run what is in begin block
      retry
    end
    self
  end

  # creates a `tag_list` instance var
  # `||=` operator is Ruby "short-circuit" which means
  # if `tag_list` is set, then skip this part,
  # if not set, then set it to what is on the right side of equals sign.
  # the purpose is to set the tag_list var equal to
  # the response from the `list_tags_for_resource`¹ 
  # which contains resource_type set to Parameter, and the 
  # resource_id set to name
  def tag_list
    @tag_list ||= client.list_tags_for_resource({resource_type: 'Parameter', resource_id: name})
  end
  # checks the `tag_list` method above & runs a 
  # select method on the tag_list hash
  # loops to see if there is a key with the `key` value in hash
  # and checks presence of a `skip_sync` tag with the `.any?` 
  # boolean method. If this exists, then the lambda function
  # will not run and the replication will not occur.
  # If this does not exist, then it proceeds. 
  # You may want to skip syncing for regional specific resources. 
  # If you want to replicate an initial skip_sync param, simply
  # remove the tag in question and on the next run, the param will sync`
  def skip_sync?
    tag_list[:tag_list].select {|key| key[:key] == $skip_tag }.any?
  end
  # Calls the Ruby `put_parameters` method on the `client_target` parameter.
  # `put_parameter` replicates name, value, type, and overwrite. This method
  # also adds the tags copied over from the tag_list method to resources by name.
  def sync_to_region(client_target)
    client_target.put_parameter({
      name: response['parameter']['name'], # required
      value: response['parameter']['value'], # required
      type: response['parameter']['type'], # accepts String, StringList, SecureString
      overwrite: true,
    })
    client_target.add_tags_to_resource({resource_type: 'Parameter', resource_id: name, tags: tag_list.to_h[:tag_list]})    
  end
end

The next file I will discuss is the ssm_full_replication.rb piece of the code. As you may gather from the name, this is responsible for full replication.

# this pulls the AWS sdk gem
require 'aws-sdk-ssm'
require 'aws-sdk-sqs'
require_relative 'parameter_store'
# Declare global variables which are set to the
# respective values from CloudFormation template.
$target_region = ENV['TARGET_REGION'] or raise "Missing TARGET_REGION variable."
$skip_tag = ENV['SKIP_SYNC'] or raise "Missing skip_sync tag."
$stage_env = ENV['STAGE_ENV']
# method set to us-east-1 for source region. 
# var `sqs_client` set to new SQS client connection in target region
# var `sts_client` set to new STS client conn in source region.
# call `send_message` on `sqs_client` var with queue_url & message_body as params.
def send_params_to_sqs(name)
  region = "us-east-1"
  sqs_client = Aws::SQS::Client.new(region: $target_region)
  sts_client = Aws::STS::Client.new(region: region)
  sqs_client.send_message(
    queue_url: "https://sqs.#{region}.amazonaws.com/#{sts_client.get_caller_identity.account}/SSM-SQS-replication-#{$stage_env}-#{region}",
    message_body: name
  )
end
# sets new SSM client connection in source region
# and new SSM client_target connection in target region
def handler(event:, context:)
  client = Aws::SSM::Client.new
  client_target = Aws::SSM::Client.new(region: $target_region)
  # next_token set to nil, which is important at start of lambda func
  next_token = nil
  # loop starts with begin block which
  # runs before the rest of the code in method.
  loop do 
    begin
      # describe_batch is set to value from 
      # describe_parameters² call on the client variable.
      @describe_batch = client.describe_parameters({
        # parameter_filter limits request results to what we need
        parameter_filters: [
          {
            key: "Type",
            values: ["String", "StringList", "SecureString"]
          },
        ],
        # next_token is set to next set of items to return
        next_token: next_token,
      })
      # describe_batch var calls iterative loop and
      # sends param name to send_params_to_sqs method
      @describe_batch.parameters.each do |item|
        send_params_to_sqs(item.name)
      end
      # break means that func will end if the next_token value is empty.
      break if @describe_batch.next_token.nil?
      next_token = @describe_batch.next_token
      # exception handling. it looks for this error message, and this is how it will handle, by pausing for 60 seconds.
    rescue Aws::SSM::Errors::ThrottlingException
      p "Sleeping for 60 seconds while describing parameters."
      sleep(60)
    end
  end
end

The last file to share is the ssm_regional_replication.rb file. This file is event based and does the regional replication.

# this pulls the AWS sdk gem
require 'aws-sdk-ssm'
require_relative 'parameter_store'
# Global vars for file
$target_region = ENV['TARGET_REGION'] or raise "Missing TARGET_REGION variable."
$skip_tag = ENV['SKIP_SYNC'] or raise "Missing skip_sync tag."
# CloudWatch sends events in a specific format compared to SQS triggered lambdas
# so this method grabs the values from CloudWatch handles both formats.
def massage_event_data(event)
  # pull out values from a cloudwatch invocation
  operation = event.fetch('detail', {})['operation']
  name      = event.fetch('detail', {})['name']
  return operation,name if operation && name
  operation = 'Update' 
  name      = event.fetch('Records', []).first['body']
  return operation,name 
end
def handler(event:, context:)
  # set vars called operation and name. output from prev. method.
  # create new client & target vars for SSM
  operation,name = massage_event_data(event)
  client = Aws::SSM::Client.new
  client_target = Aws::SSM::Client.new(region: $target_region)
  # this logic runs event based code. If the operation from 
  # the CloudWatch metrics is equal to either update or create
  # the ps var uses the ParameterStore find_by_name class method
  # and passes the client * name.
  if operation == 'Update' || operation == 'Create'
    ps = ParameterStore.find_by_name(client, name)
    # if the ps var has a skip_sync tag, then the CloudWatch logs
    # you will get what's in the puts string. if there is no tag
    # it syncs to target region.
    if ps.skip_sync?
      puts "This function has been opted out, not replicating parameter."
    else
      ps.sync_to_region(client_target)
    end
  # if the operation is delete in the source region, then the delete_parameter method is called on the
  # client_target and it's also deleted from the target_region to ensure parity.
  elsif operation == 'Delete'
    response = client_target.delete_parameter({
      name: name, # required. go into event, reference the detail key, and the value name
    })
  end
end

References to AWS API docs page:

If you want to be sure that there are no missed variables, you can always set up a CloudWatch alarm on if your lambda has any failed invocations or if your SQS queue isn't sending any messages. I hope that this has helped others who are looking for a way to replicate SSM parameters in AWS from one region to another. That's the end of the code, I know it is a lot to digest, so if you have any questions please leave a comment and I'll do my best to follow up.

Using Jekyll & AWS Amplify for your blog!

Katherine (she/her) — Fri, 08 Jan 2021 23:35:01 +0000

This blogpost will share how I deployed my Jekyll site to AWS, using AWS Amplify and Route53, be warned, it's a bit on the lengthy side.

Hi, Let's Set Up a Blog!

Jekyll is a static site generator that makes having a personal website, blog, portfolio, etc., very easy to set up. Being an Operations Engineer, my exposure to HTML & CSS really only took off back in the MySpace days, and it's been a while since I've used either. If you're looking for a low/no code solution, this may be a perfect fit for you.

What is AWS Amplify? On AWS's site they describe Amplify as a service that can help to quickly configure backends, deploy frontend static websites, and more, while supporting many frameworks.

Requirements Check

Jekyll does have a fairly robust docs page and I was able to confirm my local machine met the requirements (currently I am on MacOS 11.1, Ruby 3.0, and RubyGems 3.2.3). I actually found it easier to follow the actual blog set up process by following the README's on individual themes instead of following the default on the docs page (jekyll new myblog). Granted, not all themes are up to date and have active contributions, but the theme that I landed on did.

*** IMPORTANT FOR RUBY 3 ***

There's a known bug as of the writing of this blog post where Jekyll fails to serve on Ruby 3. Check out this open issue. TL;DR: I ran this: bundle add webrick.

I decided on the Minimal Mistakes theme and went to work using the quick start guide. First, I added this theme to my Gemfile, did a bundle install, and then added my theme to _config.yml - this was a breeze. Once that was done, I tried to confirm the base image was ok by running bundle exec jekyll serve in my terminal.

After I confirmed the out of the box config works, I pushed an initial commit to GitHub. I highly recommend using some sort of version control and committing after a big change that doesn't break your functionality. You don't want to to be put in a situation where you don't know what your last known working config was without having a good version control system.

Updating _config.yml

In my _config.yml I enabled features that I wanted, such as; Google Analytics (pro tip: set your provider to: google-gtag, didn't work for me if other providers were listed), comments, adding a photo, my domain, etc. I ran my local Jekyll server again to confirm it looked good and it did, so I moved on to creating the appropriate root files for links/nav/pages.

Links, Assets, Navigation, & Content

Added a "resume.markdown" file which points to a folder in my "assets/images/FILE_NAME" path to show an image of my resume.
Created "tag-archive.md" to set up the tags feature for better filtering of future blog posts.
Created "navigation.yml" & "ui-text.yml" in a root level directory I made called "_data" to cover the navigation for my site.
Updated the "about.markdown" file to just have a basic about me post.

Hosting on AWS

Ok! Once our blog works on our local machine & looks good, we can move on to deploying it somewhere so the rest of the world can see 😄 I decided to go with AWS Amplify for now, but may go with GitHub pages since Jekyll is hosted for free on there.

First thing I did was make sure my domain was still valid, if you do not own a domain, you can purchase one! Simply google "buy a domain" and you'll get a ton of results, do your research and make sure you find the one that best fits your needs.

Once I confirmed my domain was good, I created a Terraform file for my Route53 entry. If you are not familiar with Terraform, you can easily do this in the AWS Console instead.

SIDE NOTE

you must have an AWS account to proceed with the tutorial after this point, some features are free and fall into the free tier, some are not. PLEASE be aware of the resources you are spinning up and using, it can be a very costly cloud provider.

To create an entry in Route53 with Terraform, all you need is this:

resource "aws_route53_zone" "primary" {
  name = "example.com"
}

very easy! It creates the nameservers and other resources after doing a terraform apply.

Alternatively, if you do not want to use Terraform, in the AWS console, you can go to the Route53 page (search for this under all services) and then select the "Hosted Zones" on the left hand side. Then click "Create Hosted Zone" and enter the information requested and click the "Create Hosted Zone" button.

After a few minutes, DNS should have propagated out, take note of your "NS" (nameserver) entries, we will need these values later on for our custom domain.

AWS Amplify

In the AWS Amplify console, select "New app" & "Host web app" from the drop down. The next page will have you select a source code option, choose the provider that you have (I use GitHub).

The next page will redirect you to GitHub for authorization to your repo. Once that's done select your blog repo from the dropdown menu & branch if it's something other than main that you would like to deploy.

The next step and the final one before the review is to confirm the build settings. PRO TIP: If you are planning on using Google Analytics, edit the build in Amplify to include the JEKYLL_ENV=production before if not, it will default to the development environment, which won't push your data to Google.

To add your custom domain to AWS Route53, go to the "DNS Settings" page of your hosting provider (Google Domains, Go Daddy, etc..) and select the "Custom" option shown by your provider. Enter the nameservers exactly as shown from the AWS Route53 entry for your hosted zone and hit save.

Now, go back to the "Domain management" tab on your app in the AWS Amplify console, and select your domain that was registered in Route53 and select "Configure Domain" to the right of your domain. Make sure this option is selected: Setup redirect from https:// to https://www.

That's it! Now we have a Jekyll blog hosted on AWS Amplify using Route53. Quite a process, but it's great to expose yourself to Cloud native solutions and explore more about what AWS has to offer.

If you have any questions, comments, or feedback, please leave a comment.

Thank you for reading! :)