Forem: Allan Chua

How to Import Existing Resources in your CloudFormation Stacks

Allan Chua — Mon, 27 Nov 2023 16:35:42 +0000

Among challenges that developers and operations staff face while working on CloudFormation is the inability to easily import orphaned stateful resources (S3, DynamoDB, Aurora, OpenSearch, Elastic Search) into CloudFormation stacks.

This has been a known problem that has been plaguing developers for quite some time. Orphaned CloudFormation resources are often generated through either:

Refactoring of a CloudFormation stack that exceeded the maximum number of CloudFormation resources (500 at the time of this article's writing).
The need to segregate stateful (DynamoDB, S3, SQS, MemoryDB, RDS, etc) vs disposable resources (Lambda, API Gateway, StepFunctions) between multiple CloudFormation stacks.
The need to convert a POC project that contained both stateful and disposable compute resources in a single CloudFormation stack to comply with production compliance rules of an organization.
To regroup stateful resources into multiple independent/nested stacks
To improve deployment efficiency and reduce risk of accidental deletion of stateful resources.
The need to recover from accidental deletion of CloudFormation stacks that contained stateful resources (DynamoDB, SQS, S3 Buckets, RDS, etc) tagged with DeletionPolicy: Retain

Thankfully, AWS just published a small yet powerful update in CloudFormation ChangeSet API that enables the passing of the ImportExistingResources parameter.

As per AWS Blog,

When you deploy ChangeSets with the ImportExistingResources parameter, CloudFormation automatically imports the resources in your template that already exist in your AWS account. CloudFormation uses the custom names of resources in your template to determine their existence. With this launch, you can reduce the manual effort of import operations and avoid deployment failures because of naming conflicts.

A hands-on example:

Consider the CloudFormation template below that contains both a DynamoDB table and an S3 bucket. If you wanna test it out on your own, you may also view this Github repository to play around it.



AWSTemplateFormatVersion: 2010-09-09

Resources:
  LionDDBTable:
    Type: AWS::DynamoDB::Table
    DeletionPolicy: Retain
    Properties:
      TableName: "my-lion-ddb"
      AttributeDefinitions:
        - AttributeName: ID
          AttributeType: S
      KeySchema:
        - AttributeName: ID
          KeyType: HASH
      ProvisionedThroughput:
        ReadCapacityUnits: 5
        WriteCapacityUnits: 5

  LionS3Bucket:
    Type: AWS::S3::Bucket
    DeletionPolicy: Retain
    Properties:
      # Replace this bucket name as S3 
      # bucket names are expected to be 
      # globally unique
      BucketName: "acsg-test-lion-bucket"

If the CloudFormation stack defined above gets deleted, both the S3 bucket and the DynamoDB tables will be retained thanks to the DeletionPolicy configured with Retain strategy.

This behaviour is great if the only goal for operating these resources is to make sure that data does not get dropped maliciously. One of the downsides of deploying this strategy is that we couldn't easily import back these resources in new CloudFormation stacks.

To prove this, run the following CloudFormation stack creation script:



#!/bin/bash
aws cloudformation create-stack \
    --stack-name "petshop-stateful-stack" \
    --template-body file:///$PWD/template.cfn.yaml \
    --capabilities CAPABILITY_IAM \

After the deployment is done, you can run the following stack destruction script:



#!/bin/bash
aws cloudformation delete-stack \
    --stack-name  "petshop-stateful-stack"

Since both our S3 and DynamoDB tables are configured with DeletionPolicy: Retain settings, you can expect to still see them in their respective consoles despite the parent CloudFormation stack's deletion.

If you re-run the aws cloudformation create-stack CLI call from the previous step, you'll get the following error from CloudFormation:



Resource handler returned message: 
"Resource of type 'AWS::DynamoDB::Table' 
with identifier 'my-lion-ddb' already exists." 
(RequestToken: 016e3290-588c-09f9-a92e-497ae81f9e49, HandlerErrorCode: AlreadyExists)

IMHO, it will be a huge development boost if we can also get the --import-existing-resources flag for the aws cloudformation create-stack CLI invocation.

Using ChangeSets and ImportExistingResources Parameter

To automatically recreate the stack and import the pre-existing resources that were orphaned by the destroy step, we can create a create-change-set API call via AWS CLI and pass the --import-existing-resources parameter.



#!/bin/bash

# Define variables
stack_name="petshop-stateful-stack"
template_file="$PWD/template.cfn.yaml"
change_set_name="change-set-$(date +%s)"

# Create a change set
aws cloudformation create-change-set \
    --stack-name "$stack_name" \
    --change-set-type "CREATE" \
    --change-set-name "$change_set_name" \
    --template-body "file://$template_file" \
    --capabilities CAPABILITY_IAM \
    --import-existing-resources # Life saving parameter

# Describe the change set (optional)
aws cloudformation describe-change-set \
    --stack-name "$stack_name" \
    --change-set-name "$change_set_name"

# Execute the change set
echo "Do you want to execute this change set? (yes/no)"
read execute_decision

if [[ "$execute_decision" == "yes" ]]; then
    aws cloudformation execute-change-set \
        --stack-name "$stack_name" \
        --change-set-name "$change_set_name"
    echo "Change set executed."
else
    echo "Change set not executed."
fi

Upon the execution of the changeset, you'll be seeing "Import Complete" operations in your CloudFormation stacks.

Things to keep in mind

You'll need the latest AWS CLI version to use this feature. I'm using AWS CLI version 2.13.38 at the time of this article's writing.
The stateful resources needs to have a DeletionPolicy property configured in it and have a unique name defined on the resource.
You can't import resources into multiple CloudFormation stacks.
CloudFormation max resource count limitation (500 at the time of writing) applies to import operations.
You can use the cloudformation:ImportResourceTypes policy condition to define IAM policies that control which resource types principals can run import operations on.

Benefits

We now have the option to delete CloudFormation stacks that contains stateful resources as long as they are properly configured with DeletionPolicy: Retain property
It is now possible to easily refactor, regroup and transfer resources across new and existing CloudFormation stacks on-demand.
Less ClickOps related to importing existing resources to CloudFormation stacks.

Scaffolding a Vue 3 + Vuetify + Amplify Project (2023)

Allan Chua — Sun, 09 Apr 2023 04:10:43 +0000

This article aims to provide the most recent step by step guide on how to scaffold a base project using Vue 3, Vuetify and Amplify components. Using amplify's Vue 3 getting started tutorial was a bit challenging in my opinion as certain steps (Injection of window.global script in index.html was confusing, vite.config.js setup was lacking import alias support, directory where amplify init should be triggered isn't clearly stated) on the guide were not detailed enough to be followed by somebody completely new with using Amplify.

After reading through this article, you'd be able to:

Run npm run dev successfully without facing import issues.
Get an amplify app in your AWS account.
Have a base scaffold that could easily be extended with other core amplify components (GraphQL API, Hosting, Cognito-based Auth)

Pre-requisites

In order to follow this guide and the same exact scaffold, you will need to have the following CLIs installed on your machines.

Node 18 - I'm using the most latest version of NodeJS 18. I've installed mine using nvm (Node version manager).
amplify cli - You'll need the amplify cli to initialize the amplify app and provision backend resources later on.
aws cli - You'll need an AWS cli to configure pre-built AWS profiles that allows your machine to make changes on your AWS accounts.

Initialise the Vue 3 projects

As with any Vue 3 + Vite project, we have to scaffold the base project by using the pre-built tools provided by the Vue JS.



npm init vue@3

Install Dependencies

After the scaffolding of base components gets completed, navigate into the project's directory and install the dependencies required by the base application to run.



# Substitute this with your project name
cd ./aws-exam-trainer

npm install

Add Dependencies

We also need to install the following dependencies:

vuetify - is the most popular UI library for VueJS projects. It requires literally no design skills to get you started in crafting visually decent frontend applications.
aws-amplify - is the base library used for accessing common integrations in Amplify such as API, Auth, Analytics, etc. This library is not coupled with UI-specific frameworks such as Vue and React JS.
@aws-amplify/ui-vue - is a VueJS-specific set of cloud connected components that makes trivial tasks (Upload to S3, Cognito-backed Integration, etc) easy and fast to implement.



npm install vuetify aws-amplify @aws-amplify/ui-vue

Initialising Amplify App

The next step is to initialise the Amplify app. This is done by running the amplify init command inside the directory generated by the npm init vue@3 command from the first step.



amplify init

Running the command will present you with the following prompt and most of the configurations for vue 3 projects will be pre-filled for us. Just agree with the default configurations for now.



Note: It is recommended to run this command from the root of your app directory
? Enter a name for the project awsexamtrainer
The following configuration will be applied:

Project information
| Name: awsexamtrainer
| Environment: dev
| Default editor: Visual Studio Code
| App type: javascript
| Javascript framework: vue
| Source Directory Path: src
| Distribution Directory Path: dist
| Build Command: npm run-script build
| Start Command: npm run-script serve

? Initialize the project with the above configuration? Yes
Using default provider  awscloudformation

The next important step is to configure the AWS named profile that will be used for provisioning the Amplify app. This profile will also be used for other operations that require changes to our AWS account, such as provisioning AppSync APIs and Cognito User Pools.

In my machine, I've configured a profile called my-poc-profile and just selected this on the guided CLI provided by AWS amplify



? Select the authentication method you want to use: AWS profile

For more information on AWS Profiles, see:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

? Please choose the profile you want to use my-poc-profile
Adding backend environment dev to AWS Amplify app: dvmifg83sjtwe

Deployment completed.
Deploying root stack awsexamtrainer [ ---------------------------------------- ] 0/4
        amplify-awsexamtrainer-dev-10… AWS::CloudFormation::Stack     CREATE_IN_PROGRESS             Sun Apr 09 2023 10:08:18…     
        DeploymentBucket               AWS::S3::Bucket                CREATE_IN_PROGRESS             Sun Apr 09 2023 10:08:21…     
        AuthRole                       AWS::IAM::Role                 CREATE_IN_PROGRESS             Sun Apr 09 2023 10:08:21…     
        UnauthRole                     AWS::IAM::Role                 CREATE_IN_PROGRESS             Sun Apr 09 2023 10:08:21…     

✔ Help improve Amplify CLI by sharing non sensitive configurations on failures (y/N) · no
Deployment state saved successfully.
✔ Initialized provider successfully.
✅ Initialized your environment successfully.

Your project has been successfully initialized and connected to the cloud!

Some next steps:
"amplify status" will show you what you've added already and if it's locally configured or deployed
"amplify add <category>" will allow you to add features like user login or a backend API
"amplify push" will build all your local backend resources and provision it in the cloud
"amplify console" to open the Amplify Console and view your project status
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

Pro tip:
Try "amplify add api" to create a backend API and then "amplify push" to deploy everything

At this point, you should now have an amplify app configured on your AWS account.

Plug Vuetify and Amplify to main.js file

In order to access Vuetify UI components and Amplify base libraries, we'll have to configure the default main.js file generated by the vue init process with the following code.



import { createApp } from 'vue'
import App from './App.vue'
import router from './router'
import './assets/main.css'

// Vuetify
import 'vuetify/styles'
import { createVuetify } from 'vuetify'
import * as components from 'vuetify/components'
import * as directives from 'vuetify/directives'

// Amplify
import { Amplify } from 'aws-amplify'
import awsExports from './aws-exports'
import AmplifyVue from '@aws-amplify/ui-vue'

Amplify.configure(awsExports)

const vuetify = createVuetify({
  components,
  directives
})

const app = createApp(App)

app.use(vuetify)
app.use(AmplifyVue)
app.use(router)

app.mount('#app')

Add window.global and global exports handle (Vite Users only) to index.html

In order to make use of the UI components provided by @aws-amplify/ui-vue library to our vite-based project, we'll have to inject the snippet below that adds global property to window and a global exports object required by Amplify components.

Please inject it on the same position to avoid getting unwanted behaviours and errors later on the development. (This is one nasty mistake I've made on my initial attempts on using Amplify + VueJS projects).



<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <link rel="icon" href="/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vite App</title>
  </head>
  <body>
    <div id="app"></div>
    <!--Add this snippet-->
    <script>
      window.global = window
      var exports = {}
    </script>
    <!--End of snippet to inject-->
    <script type="module" src="/src/main.js"></script>
  </body>
</html>

Setup vite.config.js

We'll also have to configure the vite.config.js file so it would end up like this instead of the one provided by the amplify getting started docs for Vue JS.

What makes this configuration better than the one provided by the amplify team is that it allows us to import vue components from src folder without having to use relative paths.

Tip of the day
Importing components using static path syntax makes VueJS projects easier to maintain by:

Making refactoring of dependent components paths in large projects more resilient to bugs caused by relative imports failures.
Regardless of the depth of the dependent component, static path imports make it easy reference shared components by having a standard import location.
Removes the cognitive effort required from developers in mentally resolving the relative paths of components they like to import.



import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'
import { fileURLToPath, URL } from 'url'

// https://vitejs.dev/config/
export default defineConfig({
  plugins: [vue()],
  resolve: {
    alias: [
      {
        find: './runtimeConfig',
        replacement: './runtimeConfig.browser'
      },
      // This bit here is not mentioned in Amplify Docs
      { find: '@', replacement: fileURLToPath(new URL('./src', import.meta.url)) }
    ]
  }
})

For TypeScript users

For typescript users, you will have to configure the compilerOptions.skipLibCheck option to your tsconfig.json files to skip type checking of declaration files.

This can save time during compilation at the expense of type-system accuracy. For example, two libraries could define two copies of the same type in an inconsistent way. Rather than doing a full check of all d.ts files, TypeScript will type check the code you specifically refer to in your app’s source code.



...
  "compilerOptions": {
    "skipLibCheck": true,
...

Testing the Base Scaffold

At this point, you should have a basic Vue 3 project that can access both Vuetify and Amplify components without facing compile issues.



npm run dev

You should be able to see the following web page on the port allocated to vite without errors.

What's Next?

On the next articles, I'll be showcasing how to:

Install and customize a Cognito-backed login interface.
Add an AppSync API that uses at least two types of resolvers (Lambda and DynamoDB)
Automate the deployment of changes from github repos to S3+CloudFront web hosting setup.

The final project aims to generate AWS exam reviewer questions using OpenAI prompts based from a user provided link (Ideally an AWS Documentation web page).

What makes this project interesting is that AWS professionals can now generate reviewer questions in a matter of seconds, track their scores and generate reviewer content that changes over time.

Till then, see you soon.

Preventing Accidental Deletion of DynamoDBs using Table Protection Strategy Feature (March 2023)

Allan Chua — Thu, 09 Mar 2023 05:35:43 +0000

DynamoDB is the most popular NoSQL database service provided by AWS. It is considered as the de facto choice for storage of serverless-based projects because of its highly scalable, durable, and available nature.

DynamoDB traditionally offered various mechanisms that could be used to protect both the table and data contained in them which included:

Usage of IAM-based guardrails to deny delete operations on DynamoDB tables.
CloudFormation resource deletion policy which prevents CloudFormation tools from deleting your Dynamo DB tables when deleting a stack.
Point-in-time-recovery (PITR) feature which allows continuous backup of data on a second-based granularity for the past 35 days.
On-demand backup and restore which allows us to handle data restoration beyond the 35 days limitation of PITR capability.
AWS Backup for DynamoDB which allows organizations to align their backup policies and management strategies with other storage resources.

However, one feature that has been requested by end-users is "Deletion Protection" that is highly comparable with EC2 and RDS deletion protection options. The status quo changed today as AWS announced the feature as a response to customer demand.

If you visit the new DynamoDB console, you can now find a feature that enables you to turn on the deletion protection flag.

The list of tables in DynamoDB console now includes a column that indicates protection status for each DynamoDB table:

Using AWS CLI

You can also start applying protection status on new tables using CLI-based commands as the documentation on how to implement this has been published:

You would have to update your version of AWS CLI to get hold of this feature:



pip3 install --upgrade awscli

aws --version

# aws-cli/2.7.19 Python/3.9.11 Darwin/22.3.0 exe/x86_64 prompt/off

After making sure your CLI version is of at least 2.7.19, you can now run the following shell command to test out the feature:



aws dynamodb create-table </span>

    --table-name "cant-delete-db" </span>

    --attribute-definitions </span>

        AttributeName=ID,AttributeType=S </span>

        AttributeName=DeleteOpName,AttributeType=S </span>

    --key-schema </span>

        AttributeName=ID,KeyType=HASH </span>

        AttributeName=DeleteOpName,KeyType=RANGE </span>

    --provisioned-throughput </span>

        ReadCapacityUnits=5,WriteCapacityUnits=5 </span>

    --deletion-protection-enabled </span>

    --table-class STANDARD </span>

    --profile="my-poc-profile"

CloudFormation

DynamoDB CloudFormation template now supports delete protection flag. This could be done by specifying the DeletionProtectionEnabled property to true.



SampleTable:

    Type: AWS::DynamoDB::Table

    Properties:

      TableName: "my-delete-resilient-db"

      BillingMode: PAY_PER_REQUEST

      DeletionProtectionEnabled: true # Focus here

      SSESpecification:

        SSEEnabled: true

      AttributeDefinitions:

        - AttributeName: id

          AttributeType: S

      KeySchema:

        - AttributeName: id

          KeyType: HASH

Terraform

Terraform update was released 1 week later at March 17, 2023.



resource "aws_dynamodb_table" "basic-dynamodb-table" {

  name                        = "GameScores"

  billing_mode                = "PROVISIONED"

  deletion_protection_enabled = true

  hash_key                    = "UserId"

  range_key                   = "GameTitle"

attribute {

    name = "UserId"

    type = "S"

  }

attribute {

    name = "GameTitle"

    type = "S"

  }

}

CDK

At the time of this writing, there is still no or documentation updates that has been published from CDK provider.

Update March 20, 2023 there's been a new pull request in Github for DynamoDB delete protection and will be published soon in CDK and CDK DynamoDB documentation.

Update March 22, 2023 pull request was merged, still no documentation updates.

Below is a preview of how could you do it in CDK upon release.



const table = new dynamodb.Table(this, 'Table', {

  partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },

  deletionProtection: true,

});

Key Benefits

Multiple developers working on the same table: In a team environment, multiple developers may be working on the same DynamoDB table. Without proper access controls and protection strategies, there is a risk of accidental deletions or modifications that could impact the entire team's work. By using Table Protection Strategy, you can prevent accidental deletions and maintain the integrity of the table, even with multiple developers working on it.
Testing and development environments: In testing and development environments, there may be a need to create and delete tables frequently. However, this also increases the risk of accidentally deleting important data or tables. By using Table Protection Strategy, you can prevent accidental deletions and ensure that your data remains intact during the testing and development process.
Compliance requirements: Depending on your industry and regulatory requirements, you may need to have additional protections in place to prevent accidental deletions or modifications of data. By using Table Protection Strategy, you can demonstrate that you have taken appropriate measures to protect your data and comply with regulatory requirements.
Production environments: In a production environment, accidental deletions can have serious consequences, including data loss and downtime. By using Table Protection Strategy, you can prevent accidental deletions and ensure that your tables remain available and operational.

Summary

Overall, using Table Protection Strategy in DynamoDB can help you reduce the risk of data loss, maintain data integrity, and ensure compliance with regulatory requirements.

How to Retrieve All DynamoDB Tables in a Single Account using JavaScript SDK v3 (2023)

Allan Chua — Wed, 08 Mar 2023 06:02:16 +0000

In an engineering team with matured devops practices, it is often required to know all the DynamoDBs running in various accounts. This list of tables could then be used for:

Performing standard compliance checks on your table names and configuration.
Build alarm systems that monitors list of tables in a production account to make sure that every single table you need exists.
Creating backups and restoring data in DynamoDB tables.
Managing and organizing your DynamoDB tables.
Monitoring and optimizing performance of your DynamoDB tables.
Automating and integrating your application with other AWS services that use DynamoDB tables as input or output.
Auditing and tracking usage of your DynamoDB tables.

In this article, we'll be exploring how to retrieve all the DynamoDB table names using AWS SDK for JavaScript v3.

Before you start

I'd recommend using the latest version of NodeJS supported by Lambda Functions. It's also helpful to have nvm in your machine to easily switch between NodeJS versions.

# Install Node 18 to use the latest code-base
nvm install 18

node --version

# Switch to NodeJS 18
nvm use 18

You will have to install the JavaScript AWS SDKs v3 for DynamoDB. If you are using NPM, use the following commands to install the SDKs.

# If you use NPM
npm install @aws-sdk/client-dynamodb

# Install only if you want to use 
# named profiles inside your dev machine
npm install @aws-sdk/credential-providers

If you need to use yarn:

# If you use yarn
yarn add @aws-sdk/client-dynamodb

# Install only if you want to use inside your dev machine
yarn add @aws-sdk/credential-providers

The Source Code

By design, ListTablesCommand can only return a maximum of 100 DynamoDB names from a single AWS region (see the code below).

import { DynamoDBClient, ListTablesCommand } from "@aws-sdk/client-dynamodb";
import { fromIni } from "@aws-sdk/credential-providers";

const client = new DynamoDBClient({ region: "ap-southeast-1" });

// Use this code if you need named profiles
// const client = new DynamoDBClient({
//   credentials: fromIni({ profile: "my-poc-profile" }),
//   region: "ap-southeast-1",
// });

const command = new ListTablesCommand({});
const response = await client.send(command);

console.log(response);

Which would result to the following JSON response. The response below contains all the DynamoDB tables in a single region (In our case ap-southeast-1):

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'LKJEWJOI3290923902302310219',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  LastEvaluatedTableName: undefined,
  TableNames: [ 
    'dynamo-table-a', 
    'dynamo-table-b', 
    'dynamo-table-c'
   ]
}

The page limitation of 100 items per response is probably placed by AWS to gracefully handle accounts with large number of DynamoDB tables in them.

If you really want to retrieve all the tables in a single region of a single account. You can use the following code instead:

import { DynamoDBClient, ListTablesCommand } from "@aws-sdk/client-dynamodb";
import { fromIni } from "@aws-sdk/credential-providers";

const client = new DynamoDBClient({
  credentials: fromIni({ profile: "my-poc-profile" }),
  region: "ap-southeast-1",
});
let startTableName = "";
let hasResults = false;
let tableNames = [];

do {
  hasResults = false;

  const searchInput = {
    Limit: 100,
  };

  if (startTableName) {
    searchInput.ExclusiveStartTableName = startTableName;
  }

  const command = new ListTablesCommand(searchInput);
  const response = await client.send(command);

  if (response.TableNames && response.TableNames.length > 0) {
    startTableName = response.TableNames[response.TableNames.length - 1];
    tableNames = [...tableNames, ...response.TableNames];
    hasResults = true;
  }
} while (hasResults);

console.log(tableNames);

Which would result to:

[ 'dynamo-table-a', 'dynamo-table-b', 'dynamo-table-c' ]

To improve the re-usability of this code, we can export it into a modular function that looks like:

import { DynamoDBClient, ListTablesCommand } from "@aws-sdk/client-dynamodb";
import { fromIni } from "@aws-sdk/credential-providers";

export async function getAllDynamoDBTableNamesV3(profile, region) {
  const clientConfig = {};

  if (profile) {
    clientConfig.credentials = fromIni({ profile });
  }

  if (region) {
    clientConfig.region = region;
  }

  const client = new DynamoDBClient(clientConfig);
  let startTableName = "";
  let hasResults = false;
  let tableNames = [];

  do {
    hasResults = false;

    const searchInput = {
      Limit: 100,
    };

    if (startTableName) {
      searchInput.ExclusiveStartTableName = startTableName;
    }

    const command = new ListTablesCommand(searchInput);
    const response = await client.send(command);

    if (response.TableNames && response.TableNames.length > 0) {
      startTableName = response.TableNames[response.TableNames.length - 1];
      tableNames = [...tableNames, ...response.TableNames];
      hasResults = true;
    }
  } while (hasResults);

  return tableNames;
}

Now that our query could be re-used, its now easier to retrieve all Dynamo DB tables from various regions inside a single AWS account.

Single Account Multi-region Search

Let's try something that seeks all tables from different active regions:

import { getAllDynamoDBTableNamesV3 } from "./get-all-dynamodb-table-names.js";

const regions = [
  "ap-northeast-1",
  "ap-northeast-2",
  "ap-northeast-3",
  "ap-south-1",
  "ap-southeast-1",
  "ap-southeast-2",
  "ca-central-1",
  "eu-central-1",
  "eu-north-1",
  "eu-west-1",
  "eu-west-2",
  "eu-west-3",
  "sa-east-1",
  "us-east-1",
  "us-east-2",
  "us-west-1",
  "us-west-2",
];

const accountTables = await regions.reduce(async (memo, region) => {
  const regionTables = await getAllDynamoDBTableNamesV3(
    "my-poc-profile",
    region
  );
  const regionTablePair = regionTables.map((tableName) => {
    return {
      region,
      tableName,
    };
  });

  return [...(await memo), ...regionTablePair];
}, []);

console.log(accountTables);

Which should result to the following table-region key pairs:

[ 
   { 
     region: 'ap-southeast-1',
     tableName: 'dynamodb-table-a' 
   },
   { 
     region: 'ap-southeast-1',
     tableName: 'dynamodb-table-b' 
   },
   { 
     region: 'ap-southeast-1',
     tableName: 'dynamodb-table-c' 
   },
   { 
     region: 'ap-southeast-2',
     tableName: 'dynamodb-table-d' 
   },
]

How about multi-account query for all DynamoDBs in an AWS organization?

Well, I'll be leaving that for you to enjoy a bit of coding using ES6 and AWS SDK v3. To give a bit of hint, you can wrap the single AWS account query into its own module and make the profile name "Configurable" so that you can implement an parallelised search across various AWS accounts.

This approach should be helpful in monitoring several accounts in an entire organization.

AWS SAM Multi-Destination Connectors

Allan Chua — Fri, 03 Mar 2023 17:21:43 +0000

The AWS SAM CLI team has published a new feature for SAM connectors called multi-destination connector. This makes it easier to define multiple integrations to several AWS services on the same invoking resource (Source). Its primary goal is to reduce the complexities associated with maintaining and writing IAM permissions inside SAM templates.

In this article, we will be exploring:

What challenges do SAM connectors try to address.
How to define SAM multi-destination connectors.
How they improve developer experience compared to policy templates and the original variant of SAM connectors.
Some opportunities that SAM team have to improve this current version.

A little bit of context

A few months ago, SAM connectors were introduced with the following goals in mind:

Simplifying the definition of IAM permissions: With SAM connectors, developers can define IAM permissions more easily and in a more granular way.
Reducing complexity in writing and maintaining serverless applications: SAM connectors provide an abstraction layer that makes it easier to write and maintain serverless applications. They help to simplify the process of integrating multiple AWS services and reduce the need for writing complex IAM code.
Improving productivity: By simplifying the definition of IAM permissions and reducing the complexity of writing custom IAM code. This directly results to improved productivity.

However, some improvement opportunities were spotted by the developer community which included:

One connector for each source to destination pair.

The first version of SAM connector requires a single connector definition between a source and destination pair. Take the code example below which includes three dynamo db definitions, a lambda function and three SAM connectors:

Transform: AWS::Serverless-2016-10-31

Resources:
  UsersTable:
    Type: AWS::Serverless::SimpleTable

  GroupsTable:
    Type: AWS::Serverless::SimpleTable

  PoliciesTable:
    Type: AWS::Serverless::SimpleTable

  #
  # PUT Function
  #
  PutFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: nodejs18.x
      Handler: index.handler
      CodeUri: put-endpoint/
      Environment:
        Variables:
          USERS_TABLE_NAME: !Ref UsersTable
          GROUPS_TABLE_NAME: !Ref GroupsTable
          POLICIES_TABLE_NAME: !Ref PoliciesTable
      Events:
        PutResource:
          Type: "Api"
          Properties:
            Path: "/user/put"
            Method: "PUT"

  # First connector seems tolerable
  PutUserToUserDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: UsersTable
      Permissions:
        - Write

  # Until you need another one! 
  # (Shout out to multi-table design users)
  PutUserToPolicyDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: PoliciesTable
      Permissions:
        - Write

  # And yet another one
  PutUserToGroupDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: GroupsTable
      Permissions:
        - Write

With SAM multi-destination connectors, all connectors can now be grouped within their respective main resource (Original source / lambda function):

  PutFunction:
    Type: AWS::Serverless::Function
    Connectors:
     RWAccessConnector:
        Properties:
          Destination:
            - Id: UsersTable
            - Id: PoliciesTable
            - Id: GroupsTable
          Permissions:
            - Write
            - Read
    Properties:
      Runtime: nodejs18.x
      Handler: index.handler
      CodeUri: put-endpoint/
      Environment:
        Variables:
          USERS_TABLE_NAME: !Ref UsersTable
          GROUPS_TABLE_NAME: !Ref GroupsTable
          POLICIES_TABLE_NAME: !Ref PoliciesTable
      Events:
        PutResource:
          Type: "Api"
          Properties:
            Path: "/user/put"
            Method: "PUT"

More lines of code compared to Sam Policy Templates

When the first version was released, it required us to write more lines of code (9 lines for read or write connector, 10 for read and write connector) than what SAM policy templates require us to write (3 lines).

  PutUserToGroupDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: GroupsTable
      Permissions:
        - Write
        - Read

  # What it was trying to beat prior to multi-destination connector
  PutUserToGroupDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref GroupsTable

Thanks to AWS SAM connectors, the overhead code of the function scaffold can now be seen as a valuable investment. As you link more resources to the main resource, the code line savings you get from using SAM connectors scales up, resulting in increased efficiency and reduced development time.

    Connectors:
     RWAccessConnector:
        Properties:
          Destination:
            - Id: UsersTable
            - Id: PoliciesTable
            - Id: GroupsTable
            - Id: UserIdentificationCardBucket # S3
            - Id: PaymentSettlementQueue # SQS
            - Id: InvoicingTopic # SNS (Diverse resource definition on unified syntax)
          Permissions:
            - Write
            - Read

Another important advantage of AWS SAM connectors is their ability to support multiple types of destination resources using a single definition tree. This feature makes SAM connectors much more user-friendly compared to SAM policy templates.

With SAM connectors, there is no need to remember the specific property names (TableName, BucketName, QueueName, etc.) needed to link different resources, as the new syntax allows for a simplified and intuitive approach to defining resource connections

When multiple source-destination connectors are defined in separate areas of a large SAM file, it can be difficult to maintain and use them effectively.

Prior to the release of multi-destination connectors, if engineers incrementally link several resources to a function using SAM connectors, chances that they'd do it on different sections of a large SAM file are high.

Imagine engineers merging conflicting changes on a large SAM file? Sound familiar eh?

PutUserToUserDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: UsersTable
      Permissions:
        - Write

  # Imagine 200 lines in-between here

  PutUserToPolicyDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: PoliciesTable
      Permissions:
        - Write

  # Imagine 1,532 lines in-between here

  PutUserToGroupDBConnector:
    Type: AWS::Serverless::Connector
    Properties:
      Source:
        Id: PutFunction
      Destination:
        Id: GroupsTable
      Permissions:
        - Write

  # At this point, we can't remember 
  # where they link and what is their purpose don't we?

Unwanted incidents resulting from scattered resource definitions can lead to challenges in code reading for team members over the long run. To avoid these issues, it is important to establish proper standards and quality gates in the development process.

Multi-destination connectors provide a convenient solution for managing destination-specific integration code by allowing developers to place all the code inside a unified YAML node (Source Resource).

This not only streamlines the development process, but also allows for greater ease of use and more efficient resource management, making it easier to be kind to our future selves.

  PutFunction:
    Type: AWS::Serverless::Function
    Connectors:
     RWAccessConnector:
        Properties:
          Destination:
            - Id: UsersTable
            - Id: PoliciesTable
            - Id: GroupsTable
            - Id: UserIdentificationCardBucket
            - Id: PaymentSettlementQueue
            - Id: InvoicingTopic 
          Permissions:
            - Write
            - Read
   # Code trimmed for brevity

What features can SAM CLI team still improve?

Although SAM connectors offer support for a wide range of resources, they still do not cover all the resources that are available in SAM Policy Templates. It would be particularly beneficial to have connectors that support AWS Secrets Manager and AWS SSM Parameter Store, as these are two of the most commonly used features when building integrations with non-AWS resources in serverless stacks.
In my humble opinion, there is still room for improvement in SAM connectors. By implementing a more concise syntax, we can reduce the amount of boilerplate code required. For example, the following syntax could be used to define multiple connectors for a single function

  PutFunction:
    Type: AWS::Serverless::Function
    WriteConnectors:
      - Id: SampleTableA
      - Id: SampleTableB
      - Arn: arn:aws:secretsmanager:::secret:sample-credential-Lw92LOwe

This would make it easier to manage multiple connectors and simplify the development process even further.

It would be really cool if the SAM CLI team could introduce a linting rule that enforces the usage of multi-destination connectors over 1-1 destination source pairs. This would encourage developers to adopt the more efficient and streamlined approach offered by multi-destination connectors, leading to more consistent code and better resource management.

Summary

Using SAM Multi-destination connectors can improve the readability and maintainability of large SAM templates, making it easier to manage integrations between resources.
SAM Multi-destination connectors reduce integration code length, leading to more efficient code and faster development times.
SAM Multi-destination connectors group connectors by the source, leading to a more organized integration code.
We are no longer constrained to use one-to-one source to destination pair connector syntax.
There are still some opportunities to improve SAM connector syntax after this.

Wrapping Up!

I'm eager to hear your thoughts on AWS SAM Multi-destination connectors. What do you think could be improved, and how would you improve it if you had the chance to provide feedback to the SAM CLI team? Share your comments below!

Related Literature

An Introduction to Feature Toggles & AWS AppConfig

Allan Chua — Fri, 15 Apr 2022 14:45:22 +0000

Rollout changes with confidence using feature toggles.

This article’s goal is to explain feature toggles and how engineering teams can benefit from using them. It will also provide a brief introduction about AWS AppConfig and how it could be used to create and manage configuration & feature toggles dynamically during runtime.

Feature Toggling

Feature Toggles (aka Feature Flags) are used by engineering teams to dynamically enable/disable software behavior during runtime. This empowers them to continuously deploy an entire application stack which could include untested, upcoming, or features subjected for evaluation in production without actually releasing them to their consumers.

Feature toggles greatly contributes to reducing the risk of application-level failures & downtimes when new software features are rolled out to production environments. This is achieved through the dynamic enabling/disabling (Toggling) of software features during runtime using intuitive configuration management tools like AWS AppConfig.

Testing features in production is might sound crazy but there are certain situations where feature testing is only meaningful inside the context of a production environment due to the nature of traffic or amount of data in this environment.

To give an example, let’s say a shoe-string startup’s engineering team recently released an integration with a 3rd party provider’s API and they never had a chance to test the integration under stress inside staging environments due to the fact that they are using pay-per-use compute environments like AWS Lambda. Running a stress test certainly results to a sudden surge of bill and would be considered as an inefficient use of valuable compute budget.

Solving the mentioned problem requires the team to avoid load testing in staging environment to ensure efficient usage of compute budget and this could be achieved by testing the API integration inside a live production environment.

To solve this problem in an efficient and less risky manner, feature toggles could be utilized by the team to modify the application’s behavior inside production runtime by turning on the feature toggle so the integration could be included in the normal application routine.

Feature toggles enables engineering teams to observe the 3rd party API’s behavior in a live production environment and provides them the capability to disable the API integration if any throttling issues are encountered without performing a lengthy deployment.

Benefits

The use-case mentioned above is just the tip of the benefit iceberg, there are several ways in which engineering teams could take advantage of feature toggles:

Deploy invisible features with risk mitigation in-place. This is great for software features that could only be evaluated in production environments due to either the amount of traffic or data required to evaluate their working conditions.
Since engineering teams can deliver software features regardless of their validity, it would be safe for us to say that feature toggles greatly increases an engineering team’s agility.
Product managers now have the capability to run A/B tests on production without the need for complex traffic segmentation tools that often makes the production applications bloated.
Engineering teams can work on a single branch and push all code regardless if they are working or not which greatly increases an engineering team’s speed in delivering software features in production environments.
Feature toggles offer flexibility to engineering teams by providing them means of testing assumptions that couldn’t be easily proven in development environments.
Modify mobile application behaviors without resubmitting new app store and play store builds. We all know how long does it take for Apple and Google to update our mobile applications which adds another layer of complexity in testing / evaluating ideas against customer behaviors.

Use-cases

Deploy applications that requires easy customization for different countries/locales using a central code-base. This is super convenient for large multi national companies who need to design applications for different regulations and cultures of their target markets.
Enable / disable new features based on customer reactions and collected insights.
Swap out different machine learning models or alter their configurations based on observed performance in actual production environments and inferencing results they produce.
Enable/disable integration with 3rd party capabilities like Emails, SMSs, crypto wallets, analytical APIs, web hooks, etc. Dark launching of new business capabilities. (Dark launching is the capability to deploy software features without immediate unveiling them to customers)

This are the few ones that I could think of as we speak, I would be more interested in hearing your ideas and would love to have a chat with you guys in the comments section about your thoughts regarding feature toggles.

Feature Toggling Services

Feature toggling services like AWS AppConfig offers benefits that hard-coded configuration / JSON files or open source tools can’t offer:

Central management of feature flags inside easy to use GUIs.
Instant switching of features during runtime.
Eliminates the need to worry about the stability, continuity and sustainability of an open-source project.
Focus engineering efforts towards building differentiating capabilities over developing feature toggling capabilities.
You get to audit usage data of feature toggles without building analytical capabilities on your own.

AWS AppConfig

AppConfig is a service offered by AWS for the purpose of centralizing management of configuration data. It helps us store feature flag values and other configuration data so we don’t have to embed configuration data inside our codebases or build our own feature flag management tools.

AWS AppConfig also includes the capability to group related configuration through the use of application & environment constructs which makes configuration management simpler.

AWS AppConfig comes with configuration validators that acts as an additional set of safety measures geared towards reducing the number of configuration errors originating from invalid and faulty values.

The service also comes with ready-to-use deployment tool that streamlines the management of feature flags during application runtime. This tool includes several deployment strategies which makes AWS AppConfig shine better compared to other feature toggling tools out there.

It also includes a monitoring tool integrated to AWS CloudWatch that we could use to monitor and observe the impact of feature toggles without exerting too much effort.

AWS AppConfig is compatible with applications hosted on EC2, AWS Lambda, container services, mobile applications or IOT devices.

Conclusion

Feature toggles enable software engineering teams to enable/disable new software features. It empowers product owners too by providing the capability to test out assumptions and ideas without taking the risk of annoying customers in production environments. AWS AppConfig is an awesome tool that AWS customers could use to get started with feature toggling.

Stay tune for more AWS AppConfig articles

I’m currently working on a few more articles aimed at helping developers in provisioning, testing and integrating AWS AppConfig on different layers of serverless applications. If you have any use-cases that you’d like to try using AWS AppConfig, feel free to drop a comment here and let’s see how could we collaborate together to implement a proof of concept with other AWS community builders.

Further Learning

Here are some articles and videos that everybody could use to learn more about AWS AppConfig.

AppConfig User Guide [Link]
AppConfig API Reference [Link]
AppConfig Introduction Video [Link]
AppConfig Getting Started Video [Link]

Forem: Allan Chua

How to Import Existing Resources in your CloudFormation Stacks

A hands-on example:

Using ChangeSets and ImportExistingResources Parameter

Things to keep in mind

Benefits

Related Links:

Scaffolding a Vue 3 + Vuetify + Amplify Project (2023)

Pre-requisites

Initialise the Vue 3 projects

Install Dependencies

Add Dependencies

Initialising Amplify App

Plug Vuetify and Amplify to main.js file

Add window.global and global exports handle (Vite Users only) to index.html

Setup vite.config.js

For TypeScript users

Testing the Base Scaffold

What's Next?

Preventing Accidental Deletion of DynamoDBs using Table Protection Strategy Feature (March 2023)

Using AWS CLI

CloudFormation

Terraform

CDK

Key Benefits

Summary

How to Retrieve All DynamoDB Tables in a Single Account using JavaScript SDK v3 (2023)

Before you start

The Source Code

Single Account Multi-region Search

How about multi-account query for all DynamoDBs in an AWS organization?

AWS SAM Multi-Destination Connectors

A little bit of context

One connector for each source to destination pair.

More lines of code compared to Sam Policy Templates

When multiple source-destination connectors are defined in separate areas of a large SAM file, it can be difficult to maintain and use them effectively.

What features can SAM CLI team still improve?

Summary

Wrapping Up!

Related Literature

An Introduction to Feature Toggles & AWS AppConfig

Feature Toggling

Benefits

Use-cases

Feature Toggling Services

AWS AppConfig

Conclusion

Stay tune for more AWS AppConfig articles

Further Learning