Forem: Rafal Wilinski

I built ChatWithCloud - Talk with your AWS Cloud from Terminal, literally

Rafal Wilinski — Tue, 02 Jan 2024 14:24:00 +0000

Hi Developers 👋

Lets face it, the world of AWS with +200 services, thousands of SDK/CLI calls, IAM permissions & cryptic error messages can be really hard. It's an incredibly powerful tool but sometimes just too overwhelming.

Seeing an explosion of "Chat with your PDF" apps, I realised that similar solution for AWS could be really helpful.

🚀 That's how ChatWithCloud was born.

// Detect dark theme var iframe = document.getElementById('tweet-1742183912141472192-663'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1742183912141472192&theme=dark" } .

Example use cases of ChatWithCloud include:

Cost Analysis - Find out how much you are spending on AWS and what you can do to optimize your costs.
Security Analysis - Find your resources, analyze your IAM policies, improve your security posture.
Troubleshooting - Understand your infrastructure & its issues, find out what is wrong and how to fix it.
Fixing stuff - ChatWithCloud can help you fix your infrastructure, and it will do it for you.

ChatWithCloud is available through NPM: npx chatwithcloud. You can learn more about it on our homepage.

Let me know what do you think!

Show dev.to: CMD + K for AWS Console extension

Rafal Wilinski — Thu, 29 Sep 2022 10:53:24 +0000

Hi everyone,

How many times you felt lost in the AWS Console? All the services, regions, and suboptimal UI make using AWS troublesome.

Built-in search also doesn't help. Often in returns irrelevant results and most importantly, it's not context aware, it knows nothing about your resources.

I decided to solve this problem and created this:

✨ CloudTempo - Fast & Smart Command Bar for AWS Console ✨

// Detect dark theme var iframe = document.getElementById('tweet-1569356215565565952-260'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1569356215565565952&theme=dark" }

CloudTempo lets you:

Quickly find your AWS resources:

Even if you're not sure what's the name; fuzzy matching included:

Use query language to precisely find that pesky resource:

And much more!

CloudTempo is available for Chrome and MS Edge.

Learn more about it here

DynamoDB Streams - The Ultimate Guide

Rafal Wilinski — Sat, 05 Mar 2022 20:42:04 +0000

What are DynamodDB Streams? How do they work?

DynamoDB Stream can be described as a stream of observed changes in data, technically called a Change Data Capture (CDC). Once enabled, whenever you perform a write operation to the DynamoDB table, like put, update or delete, a corresponding event containing information like which record was changed and what was changed will be saved to the Stream in near-real time.

Characteristics of DynamoDB Stream

events are stored up to 24 hours
ordered, sequence of events in the stream reflects the actual sequence of operations in the table
near-real time, events are available in the stream within less than a second from the moment of the write operation
deduplicated, each modification corresponds to exactly one record within the stream
noop operations, like PutItem or UpdateItem that do not change the record are ignored

Anatomy of DynamoDB Stream

Stream consists of Shards. Each Shard is a group of Records, where each record corresponds to a single data modification in the table related to that stream.

Shards are automatically created and deleted by AWS. Shards also have a possibility of dividing into multiple shards, and this also happens without our action.

Moreover, when creating a stream you have few options on what data should be pushed to the stream. Options include:

OLD_IMAGE - Stream records will contain an item before it was modified
NEW_IMAGE - Stream records will contain an item after it was modified
NEW_AND_OLD_IMAGES - Stream records will contain both pre and post-change snapshots
KEYS_ONLY - Self-explanatory

DynamoDB Lambda Trigger

DynamoDB Streams works particularly well with AWS Lambda due to its event-driven nature. They scale to the amount of data pushed through the stream and streams are only invoked if there's data that needs to be processed.

In Serverless Framework, to subscribe your Lambda function to a DynamoDB stream, you might use following syntax:

functions:
  compute:
    handler: handler.compute
    events:
      - stream: arn:aws:dynamodb:region:XXXXXX:table/foo/stream/1970-01-01T00:00:00.000
      - stream:
          type: dynamodb
          arn:
            Fn::GetAtt: [MyDynamoDbTable, StreamArn]
      - stream:
          type: dynamodb
          arn:
            Fn::ImportValue: MyExportedDynamoDbStreamArnId

Sample event that will be delivered to your Lambda function:

[
  {
    "eventID": "fe981bbed304aaa4e666c0ecdc2f6666",
    "eventName": "MODIFY",
    "eventVersion": "1.1",
    "eventSource": "aws:dynamodb",
    "awsRegion": "us-east-1",
    "dynamodb": {
      "ApproximateCreationDateTime": 1576516897,
      "Keys": {
        "sk": {
          "S": "sk"
        },
        "pk": {
          "S": "pk"
        }
      },
      "NewImage": {
        "sk": {
          "S": "sk"
        },
        "pk": {
          "S": "pk"
        },
        "List": {
          "L": [
            {
              "S": "FirstString"
            },
            {
              "S": "SecondString"
            }
          ]
        },
        "Map": {
          "M": {
            "Name": {
              "S": "Rafal"
            },
            "Age": {
              "N": "30"
            }
          }
        },
        "IntegerNumber": {
          "N": "223344"
        },
        "String": {
          "S": "Lorem Ipsum"
        },
        "StringSet": {
          "SS": ["Test1", "Test2"]
        }
      },
      "SequenceNumber": "125319600000000013310543218",
      "SizeBytes": 200,
      "StreamViewType": "NEW_IMAGE"
    },
    "eventSourceARN": "arn:aws:dynamodb:us-east-1:1234567890:table/my-test-table/stream/2021-12-02T00:00:00.000"
  }
]

Few important points to note about this event:

eventName property can be one of three values: INSERT, MODIFY or REMOVE
dynamodb.Keys property contains the primary key of the record that was modified
dynamodb.NewImage property contains the new values of the record that was modified. Keep in mind that this data is in DynamoDB JSON format, not plain JSON. You can use library like dynamodb-streams-processor by Jeremy Daly to convert it to plain JSON.

Filtering DynamoDB Stream events

One of the recently announced features of Lambda function is the ability to filter events, including these coming from a DynamoDB Stream. Filtering is especially useful if you want to process only a subset of the events in the stream, e.g. only events that are deleting records or updating a specific entity. This is also a great way to reduce the amount of data that your Lambda function processes - it drives the operational burden and costs down.

To create event source mapping with filter criteria using AWS CLI, use following command:

aws lambda create-event-source-mapping \
--function-name dynamodb-async-stream-processor \
--batch-size 100 \
--starting-position LATEST \
--event-source-arn arn:aws:dynamodb:us-west-2:111122223333:table/MyTable/stream/2021-05-11T12:00:00.000 \
--filter-criteria '{"Filters": [{"Pattern": "{\"age\": [{\"numeric\": [\"<\", 25]}]}"}]}'

The filter-criteria argument is using the same syntax as EventBridge event patterns.

DynamoDB Streams Example Use Cases

DynamoDB Streams are great if you want to decouple your application core business logic from effects that should happen afterward. Your base code can be minimal while you can still "plug-in" more Lambda functions reacting to changes as your software evolves. This enables not only separation of concerns but also better security and reduces the impact of possible bugs. Streams can be also leveraged to be an alternative for Transactions is consistency and atomicity is not required.

Data replication

Even though cross-region data replication can be solved with DynamoDB Global tables, you may still want to replicate your data to DynamoDB table in the same region or push it to RDS or S3. DynamoDB Streams are perfect for that.

Content moderation

DynamoDB Streams are also useful for writing "middlewares". You can easily decouple business logic with asynchronous validation or side-effects. One example of such a case is content moderation. Once a message or image is added to a table, DynamoDB Stream passes that record to the Lambda function, which validates it against AWS Artificial Intelligence services such as AWS Rekognition or AWS Comprehend.

Search

Sometimes the data must also be replicated to other sources, like Elasticsearch where it could be indexed in order to make it searchable. DynamoDB Streams allow that too.

Best practices

Be aware of the eventual consistency of that solution. DynamoDB Stream events are near-real time but not real-time. There will be small delay between the time of the event and the time of the event delivery.
Be aware of constraints - events in the stream are retained for 24 hours, only two processes can be reading from a single stream shard at a time
To achieve best separation of concerns, use one Lambda function per DynamoDB Stream. It will help you keep IAM permissions minimal and code as simple as possible
Handle failures. Wrap whole processing logic in a try/catch clause, store the failed event in a DLQ (Dead Letter Queue) and retry them later.

Notifications and sending e-mails

Similarly to the previous example, once the message is saved to DynamoDB table, Lambda function which subscribes to that stream, invokes AWS Pinpoint or SES to notify recipients about it.

Frequently Asked Questions

How much do DynamoDB streams cost?

DynamoDB Streams are based on "Read Request Units" basis. To learn more about them head to our DynamoDB Pricing calculator.

How can I view DynamoDB stream metrics?

DynamoDB Stream metrics can be viewed in two places:

In AWS DynamoDB Console in the Metrics tab
Using AWS Cloudwatch

What are DynamoDB Stream delivery guarantees?

DynamoDB Streams operate in exactly-once delivery mode meaning that for each data modification, only one event will be delivered to your subscribers.

Can I filter DynamoDB Stream events coming to my Lambda function?

Yes, you can use "event source mapping" with a filter criteria to filter events coming to your Lambda function. It's using the same syntax as EventBridge event patterns.

DynamoDB in Python's Boto3 - Query Cheat Sheet with Examples

Rafal Wilinski — Sat, 05 Mar 2022 20:38:19 +0000

What is Boto3?

Boto3 is a Python library for AWS (Amazon Web Services), which helps interacting with their services including DynamoDB - you can think of it as DynamoDB Python SDK. It empowers developers to manage and create AWS resources and DynamoDB Tables and Items.

If you're looking for similar guide but for Node.js, you can find it here, for Java, for Rust, and for Golang / Go here.

List of DynamoDB Boto3 Query Examples

Connecting Boto3 to DynamoDB
Create Table
Get All Items / Scan
Get Item
Batch Get Item
Put Item
Query Set of Items
Update Item
Conditionally Update Item
Increment Item Attribute
Delete Item
Delete All Items
Query with Sorting
Query Pagination
Run DynamoDB Local

Connecting Boto3 to DynamoDB

Connecting to DynamoDB with boto3 is simple if you want to do that using Access and Secret Key combination:

import boto3

client = boto3.client('dynamodb',
  aws_access_key_id='yyyy',
  aws_secret_access_key='xxxx',
  region_name='us-east-1')

Keep in mind that using access and secret keys is against best security practices, and you should instead use IAM roles/policies to interact with DynamoDB. Boto3, if ran on Lamba function or EC2 instance, will automatically consume IAM Role attached to it.

Create Table with Boto3

DynamoDB structures data in tables, so if you want to save some data to DynamoDB, first you need to create a table. You can do that using AWS Console, AWS CLI or using boto3, like this:

import boto3

dynamodb = boto3.resource('dynamodb', region_name='us-west-2')


table = dynamodb.create_table(
    TableName='Movies',
    KeySchema=[
        {
            'AttributeName': 'year',
            'KeyType': 'HASH'  #Partition key
        },
        {
            'AttributeName': 'title',
            'KeyType': 'RANGE'  #Sort key
        }
    ],
    AttributeDefinitions=[
        {
            'AttributeName': 'id',
            'AttributeType': 'N'
        },
        {
            'AttributeName': 'createdAt',
            'AttributeType': 'S'
        },

    ],
    ProvisionedThroughput={
        'ReadCapacityUnits': 10,
        'WriteCapacityUnits': 10
    }
)

print("Table status:", table.table_status)

Keep in mind that provisioning a table takes some before it's active. If you want to know when it's ready to be used, you can use waiter function.

import botocore.session

session = botocore.session.get_session()
dynamodb = session.create_client('dynamodb', region_name='us-east-1') # low-level client

waiter = dynamodb.get_waiter('table_exists')
waiter.wait(TableName="my-table-name")

Boto3 Get All Items aka Scan

To get all items from DynamoDB table, you can use Scan operation. The problem is that Scan has 1 MB limit on the amount of data it will return in a request, so we need to paginate through the results in a loop.

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.scan()
data = response['Items']

while 'LastEvaluatedKey' in response:
    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
    data.extend(response['Items'])

You can apply FilterExpression attribute in order to filter the results like this:

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.scan(FilterExpression=Attr('country').eq('US') & Attr('city').eq('NYC'))
data = response['Items']

while 'LastEvaluatedKey' in response:
    response = table.scan(ExclusiveStartKey=response['LastEvaluatedKey'])
    data.extend(response['Items'])

Boto3 Get Item

To get a single item from DynamoDB using Partition Key (and Sort Key if using composite key), you can use GetItem operation.

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.get_item(Key={
  primaryKeyName: "ID-1",
  sortKeyName: "SORT_2"
})

Keep in mind to replace primaryKeyName and sortKeyName with actual keys from your table.

Boto3 Batch Get Item

If you want to retrieve multiple items identified by a key(s) in one call, use batch_get_item call with the following syntax:

import boto3

dynamodb = boto3.resource('dynamodb', region_name=region)

response = dynamodb.batch_get_item(
        RequestItems={
            'my-table': {
                'Keys': [
                    {
                        'id': 1
                    },
                    {
                        'id': 2
                    },
                ],
                'ConsistentRead': True
            }
        },
        ReturnConsumedCapacity='TOTAL'
    )

Keep in mind that batch_get_item is limited to 100 items and 16 MB of data.

Boto3 Put Item

To write a single item into the DynamoDB Table, use PutItem operation:

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.put_item(
    Item={
        'id': 1,
        'title': 'my-document-title',
        'content': 'some-content',
    }
)

Boto3 Query for a set of items

Alternative way to get a collection of items is the Query method. Query is much faster than Scan because it uses Indexes. It should be your preferred way to get a collection of items with the same partition key.

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.query(
    KeyConditionExpression=Key('id').eq(1)
)

for i in response['Items']:
    print(i['title'], ":", i['description'])

Keep in mind that Query can return up to 1MB of data and you can also use FilterExpressions here to narrow the results on non-key attributes.

If you don't know how to construct your Query, use Dynobase with Query Code Generation feature which will automatically generate it for you.

Boto3 Update Item

DynamoDB update_item operation consists of three primary attributes:

Key - which object should be updated
ExpressionAttributeValues - map with new values
UpdateExpression - how these new values should be applied to the object in the table

They can be used like this:

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.update_item(
    Key={
        'id': '894673'
    },
    UpdateExpression='SET country = :newCountry",
    ExpressionAttributeValues={
        ':newCountry': "Canada"
    },
    ReturnValues="UPDATED_NEW"
)

Boto3 Conditionally Update Item

Moreover, you can also add a ConditionExpression parameter, which restricts the update logic only if the evaluated expression equals true.

import boto3

dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.update_item(
    Key={
        'id': '894673'
    },
    UpdateExpression='SET country = :newCountry",
    ConditionExpression='attribute_not_exists(deletedAt)' # Do not update if deleted
    ExpressionAttributeValues={
        ':newCountry': "Canada"
    },
    ReturnValues="UPDATED_NEW"
)

Boto3 Increment Item Attribute

Incrementing a Number value in DynamoDB item can be achieved in two ways:

Fetch item, update the value with code and send a Put request overwriting item
Using update_item operation.

While it might be tempting to use first method because Update syntax is unfriendly, I strongly recommend using second one because of the fact it's much faster (requires only one request) and atomic (imagine value updated by other client after you fetched item).

To do that using single update_item operation, use following syntax:

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.update_item(
    Key={
        'id': '777'
    },
    UpdateExpression='SET score.#s = score.#s + :val",
    ExpressionAttributeNames={
        "#s": "goals"
    },
    ExpressionAttributeValues={
        ':val': decimal.Decimal(1)
    },
    ReturnValues="UPDATED_NEW"
)

Boto3 Delete Item

Deleting a single item from DynamoDB table is similar to GetItem operation. Key argument accepts primary key and sort/range key if table has composite key.

import boto3
dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.table.delete_item(Key={
    primaryKeyName: primaryKeyValue
})

Boto3 Delete All Items

Unfortunately, there's no easy way to delete all items from DynamoDB just like in SQL-based databases by using DELETE FROM my-table;. To achieve the same result in DynamoDB, you need to query/scan to get all the items in a table using pagination until all items are scanned and then perform delete operation one-by-one on each record.

import boto3

dynamodb = boto3.resource('dynamodb', region_name=region)
table = dynamodb.Table('my-table')

with table.batch_writer() as batch:
     # Iterate through table until it's fully scanned
    while scan is None or 'LastEvaluatedKey' in scan:
        if scan is not None and 'LastEvaluatedKey' in scan:
            scan = table.scan(
                ProjectionExpression='yourPrimaryKey', # Replace with your actual Primary Key
                ExclusiveStartKey=scan['LastEvaluatedKey'],
            )
        else:
            scan = table.scan(ProjectionExpression='yourPrimaryKey')

        for item in scan['Items']:
            batch.delete_item(Key={'yourPrimaryKey': item['yourPrimaryKey']})

Fortunately, this is possible just with 3 clicks using Dynobase.

Boto3 Query with Sorting

Unfortunately, DynamoDB offers only one way of sorting the results on the database side - using the sort key. If your table does not have one, your sorting capabilities are limited to sorting items in application code after fetching the results. However, if you need to sort DynamoDB results on sort key descending or ascending, you can use following syntax:

import boto3

dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.query(
    ScanIndexForward=False # true = ascending, false = descending
)
data = response['Items']

Boto3 Query Pagination

Similar to Scan operation, Query returns results up to 1MB of items. If you need to fetch more records, you need to issue a second call to fetch the next page of results. If LastEvaluatedKey was present in response object, this table has more items like requested and another call with ExclusiveStartKey should be sent to fetch more of them:

import boto3

dynamodb = boto3.resource('dynamodb', region_name=region)

table = dynamodb.Table('my-table')

response = table.query()
data = response['Items']

# LastEvaluatedKey indicates that there are more results
while 'LastEvaluatedKey' in response:
    response = table.query(ExclusiveStartKey=response['LastEvaluatedKey'])
    data.update(response['Items'])

Using Boto3 with DynamoDB Local/Offline

If you need to use DynamoDB offline locally, you can use DynamoDB local distributed by AWS or DynamoDB from Localstack. Connecting to it is as easy as changing the endpoint parameter in boto3.resource call.

import boto3

# Use port 8000 for DynamoDB Local and 4569 for DynamoDB from LocalStack
dynamodb = boto3.resource('dynamodb',
                          region_name=region,
                          endpoint_url='http://localhost:8000')

# ... rest of your code

Learn more about running and using DynamoDB locally.

DynamoDB Transactions - The Ultimate Guide (w/ Examples)

Rafal Wilinski — Sat, 05 Mar 2022 20:36:07 +0000

DynamoDB supports classical ACID transactions. But let's start with the basics.

What is ACID Transaction?

Wikipedia says:

A database transaction symbolizes a unit of work performed within a database management system (or similar system) against a database and treated coherently and reliably independent of other transactions. A transaction generally represents any change in a database.

Let me explain a "transaction" to you using more human terms:

Imagine it's Saturday. You're going to the bar with a strict entry policy with two of your friends. As you approach the bar, the bouncer looks at your first friend and lets him in. Then, he looks at you and lets you in. Finally, he looks at your second friend and says: "Sorry, it's not your day", and he is not letting him in. Because you are good friends and you always support each other, in the act of solidarity, you and your friend who managed to get in decide to leave the bar. One for all, all for one, like Three Musketeers. In transaction terms, you decided to "rollback".

Next Saturday, you're going to the same bar, but this time you got lucky, and all three of you managed to get in, and nobody needs to leave. Congrats! In the transactions world, this is a "committed" transaction.

Transactions in database world have four basic properties:

Atomicity: Guarantees that a transaction is either fully committed or not at all. Each transaction is a single unit of change to the database and it is either committed or rolled back.
Consistency: Guarantees that the database is in a consistent state after a transaction is committed.
Isolation: Guarantees that a transaction is executed in a way that does not affect other transactions.
Durability: Guarantees that a transaction is committed even if the system fails.

Translating this story to a more technical jargon

Transactions are groups of read or write operations performed atomically (multiple statements are treated as one), consistently (state after transaction will be valid), in insolation (ensures concurrent transactions are executed like sequentially) and durable (once the transaction has been committed, you can be sure that it has been persisted to the table).

Why Transactions are important

Transactions are super useful when dealing with mission-critical data like currency, item stocks, or, generally speaking - resources. Imagine implementing a banking system, sending money from account A to B. In your first naive implementation, you first subtract money from account A, and then add it to account B. It might sound totally correct.

However, there is a chance that your application will crash between the first and second operation. Or that second operation was rejected. This will render your application state corrupted, and a portion of the money would be lost in-flight.

Transactions prevent problems like this one from happening. In your application, the operation of both subtracting money from account A, and adding it to the account B, would be treated as one indivisible call. There's no possibility of your code crashing between calls, and if the second operation is rejected, the first calculation is rollbacked too. It's always all-or-nothing.

Other use cases for transactions include:

Maintaining uniqueness
Checking idempotency
Restricting actions depending on permissions

How to do transactions in DynamoDB using Node.js

Since we already know why it is important, the chances are that you've identified such patterns in your application, and you would like to implement them. Here's how you can implement a simple transaction against DynamoDB using Node.js:

const AWS = require("aws-sdk");
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region:'us-east-1' });

await dynamoDB.transactWrite({
  TransactItems: [
    {
      Put: { // Write an item to the Cart table
        Item: { // Actual Item
          id: '777',
          count: '1',
          description: 'Lorem ipsum...'
        }
        TableName: "Cart",
      },
    },
    {
      Update: { // Decrement count of items available to buy only if the count is greater than zero
        ConditionExpression: "#count > 0",
        ExpressionAttributeNames: { "#count": "count" },
        ExpressionAttributeValues: {
         ":value": 1,
        },
        Key: {
         id: '777-000',
        }
        TableName: "Items",
        UpdateExpression: "SET #count = :count - :value",
        },
      },
    ],
  }).promise();

Let's break it down

To create a transaction in DynamoDB, you can use documentClient.transactWrite.
0 It takes a parameter with TransactItems property - an Array of operations that should be performed.

Each of this array items must have a one of top-level property:
- Put - Inserts an item into the table. Use the same syntax as in put operation
- Update - Updates an item from the table. Use the same syntax as in update operation. Inside it, you can write ConditionExpressions to update item conditionally. If condition is not met, whole transaction is rollbacked.
- Delete - Deletes an item from the table. Use the same syntax as in delete operation
- ConditionCheck - A condition to an item that is not being modified by the transaction. Like in Update, if this one is not met, whole transaction is rejected.
To use modern ES syntax, whole expression is ended with a .promise() call and awaited on.

This illustrates a write transaction. What if you wanted to read items using transaction? Well, that's also possible.

Error handling

Transactions can be cancelled for a multiple reasons:

ConditionalCheckFailed - the ConditionExpression in one of the items wasn't met so the whole transaction is rolled back
TransactionConflict - the transaction was rejected because of a conflict with another ongoing transaction
ValidationError - the transaction was rejected because of a validation error, someparameters might be invalid, some updates might update keys beyond allowed limits, item size might exceed 400Kb, there might be a mismatch between operand in the update expression and the type, and so on.
TransactionInProgressException - the transaction was rejected because another transaction with the same request token is already in progress

Limits and Caveats

Transaction operations have the same limitations as they have in a normal form. For example, the size of the item inserted must be smaller than 400KB.
Total size of all the items used in the transaction must be smaller than 4MB.
If there's another transaction in progress updating the same item, any other transactions will be rejected.
TransactItems can only have up to 25 operations. Note that they can target multiple tables.
Each of the TransactItems operations can use a different table! Yes, you can do multi-table transactions.
Transactions are not incurring any additional costs.

AWS DynamoDB DocumentClient & Node.js - Complete Cheat Sheet

Rafal Wilinski — Sun, 30 May 2021 15:13:28 +0000

DynamoDB Node.js Query Examples

This cheat sheet should help you how to perform variefy of operations starting from simple queries ending with complex transactions using AWS DynamoDB DocumentClient and Node.js.

There are two basic ways to interact with DynamoDB tables from Node.js applications:

Class AWS.DynamoDB from AWS SDK for JavaScript/Typescript
AWS.DynamoDB.DocumentClient which simplifies working with DynamoDB items by abstracting away DynamoDB Types and converting responses to native JS

This cheat sheet will mostly focus on DocumentClient but some of the operations like creating tables must be run using classical DynamoDB service.

If you're looking for similar cheat sheet but for Python, you can find it here

Bonus: AWS recently announced Node.js AWS SDK v3. I wrote a tutorial on how to use new SDK V3, especially in the DynamoDB context

Setting up
Create Table
Delete Table
List Tables
Get All Items / Scan
Get Item
Batch Get Item
Put Item
Batch Write Item
Query for a Set of Items
Query an Index
Update Item
Conditionally Update Item
Increment Item Attribute
Delete Item
Delete All Items
Simple Transaction
Read Transaction
Query with Sorting
Query Pagination
Run DynamoDB Local

Setup

Setting up your Node.js application to work with DynamoDB is fairly easy. First, make sure that aws-sdk is installed, either by running yarn add aws-sdk or npm i aws-sdk --save. Then, paste following piece of code:

const AWS = require("aws-sdk") // Or use `import` syntax for Typescript and newer ES versions

const dynamoDB = new AWS.DynamoDB({
  region: "us-east-1", // If not set, will get from ~/.aws directory or environment variable
  // and rest of properties
})

// Or

const documentClient = new AWS.DynamoDB.DocumentClient({
  region: "us-east-1",
  // and rest of properties
})

As one of the Node.js best practices is to avoid callback hell, we'll be ending up all of our SDK calls with .promise() call in order to get Promises returned from SDK. It will also allow us to use async/await syntax which makes our programs much more readable.

Keep in mind that using access and secret keys is against best security practices, and you should instead use IAM roles/policies to interact with DynamoDB. This code, if ran on Lamba function or EC2 instance, will automatically use IAM Role attached to it.

Create table

DynamoDB structures data in tables, so if you want to save some data to DynamoDB, first you need to create a table. You can do that using AWS Console, AWS CLI or using AWS-SDK for Javascript, like this:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB({ region: "us-east-1" })

dynamoDB
  .createTable({
    AttributeDefinitions: [
      {
        AttributeName: "id",
        AttributeType: "S",
      },
    ],
    KeySchema: [
      {
        AttributeName: "id",
        KeyType: "HASH",
      },
    ],
    BillingMode: "PAY_PER_REQUEST",
    TableName: "my-table",
  })
  .promise()
  .then(data => console.log("Success!", data))
  .catch(console.error)

After this call resolves, it does not necessarily mean that table status is ACTIVE and it's is ready for read and write operations. Before start manipulating items in it, we should check if it's in ACTIVE state first using describeTable function ran every 5 seconds until it is ready:

const backoffInterval = 5000 // 5 seconds

const waitForTable = TableName =>
  dynamoDB
    .describeTable({ TableName })
    .promise()
    .then(data => {
      if (data.Table.TableStatus !== "ACTIVE") {
        console.log(
          `Table status: ${data.Table.TableStatus}, retrying in ${backoffInterval}ms...`
        )
        return new Promise(resolve => {
          setTimeout(() => waitForTable().then(resolve), backoffInterval)
        })
      } else {
        return
      }
    })
    .catch(error => {
      console.warn(
        `Table not found! Error below. Retrying in ${backoffInterval} ms...`,
        error
      )

      return new Promise(resolve => {
        setTimeout(() => waitForTable().then(resolve), backoffInterval)
      })
    })

waitForTable("my-table").then(() => console.log(`my-table is ready!`))

Delete table

If you changed your mind and need to remove DynamoDB table, don't worry, it's simple:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB({ region: "us-east-1" })

dynamoDB
  .deleteTable({
    TableName: "my-table",
  })
  .promise()
  .then(() => console.log("Table has been deleted"))
  .catch(console.error)

Keep in mind that Dynobase is capable to removing tables too.

List Tables

If you want to check what tables are available at your disposal in current region, use listTables call. Keep in mind that if selected region has more than 100 tables you'll have to paginate through them to fetch a complete list.

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB({ region: "us-east-1" })

dynamoDB
  .listTables()
  .promise()
  .then(() => console.log("Table has been deleted"))
  .catch(console.error)

Get All Items / Scan in DynamoDB

After our table is provisioned and it's in ACTIVE state, first thing that we probably would like to do is get all items in it aka use (DynamoDB Scan operation):

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient()

dynamoDB
  .scan({
    TableName: "my-table",
  })
  .promise()
  .then(data => console.log(data.Items))
  .catch(console.error)

If you want to narrow your search results, use FilterExpressions combined with ExpressionAttributeNames object like so:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: "us-east-1" })

dynamoDB
  .scan({
    TableName: "my-table",
    FilterExpression:
      "attribute_not_exists(deletedAt) AND contains(firstName, :firstName)",
    ExpressionAttributeValues: {
      ":firstName": "John",
    },
  })
  .promise()
  .then(data => console.log(data.Items))
  .catch(console.error)

You can find full reference how to write FilterExpressions in this post by Alex Debrie.

The snippet above will in fact return all the items in the table under one condition - you have less than 1MB of data inside it. If your table is bigger than that, you'll have to run Scan command a few times in a loop using pagination.

Get Item

If you know the exact Partition Key (and Sort Key if using composite key) of the item that you want to retrieve from the DynamoDB table, you can use get operation:

const AWS = require("aws-sdk")
AWS.config.update({ region: "us-east-1" })
const dynamoDB = new AWS.DynamoDB.DocumentClient()

dynamoDB
  .get({
    TableName: "my-table",
    Key: {
      id: "123", // id is the Partition Key, '123' is the value of it
    },
  })
  .promise()
  .then(data => console.log(data.Item))
  .catch(console.error)

Batch Get Item

DocumentClient is also capable of running bunch of get operations in a single call to the DynamoDB service:

const AWS = require("aws-sdk")
AWS.config.update({ region: "us-east-1" })
const dynamoDB = new AWS.DynamoDB.DocumentClient()

dynamoDB
  .batchGet({
    RequestItems: {
      "my-table": {
        Keys: [
          {
            id: "123",
          },
          {
            id: "124",
          },
        ],
      },
      "other-table": {
        Keys: [
          {
            id: "abc",
          },
          {
            id: "abd",
          },
        ],
      },
    },
  })
  .promise()
  .then(data => console.log(data.Responses))
  .catch(console.error)

As you can see, the RequestItems objects can accept multiple table names and can fetch multiple items from multiple tables in a single call. Keep in mind that number of items retrieved using batchGet is limited to 100 items or 16MB of data.

Moreover, if you exceed table capacity, this call will return UnprocessedKeys attribute containing a map of keys which weren't fetched.

Put Item aka Write

put operation creates a new item, or replaces an old item with a new item if it's using the same key(s):

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: "us-east-1" })

dynamoDB
  .put({
    Item: {
      id: "12346",
      name: "John Doe",
      email: "john@doe.io",
    },
    TableName: "my-table",
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

Batch Write / Put Item

If you need to insert, update or delete multiple items in a single API call, use batchWrite operation. It bundles multiple database requests against multiple tables into a single SDK call. It decreases amount of network calls needed to be made, reduces the overall latency and makes your application faster.

Example request deleting one item with key id = 123 and putting another to the same table:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: "us-east-1" })

dynamoDB
  .batchWrite({
    RequestItems: {
      MyTable: [
        {
          DeleteRequest: {
            Key: { id: "123" },
          },
        },
        {
          PutRequest: {
            Item: {
              id: "234",
              name: "dynobase",
              email: "dynobase@dynobase.dev",
              description: "Professional DynamoDB Client",
            },
          },
        },
      ],
    },
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

If you want to write into multiple tables at once, simply specify these tables inside RequestItems object like this:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: "us-east-1" })

dynamoDB
  .batchWrite({
    RequestItems: {
      TableOne: [
        {
          DeleteRequest: {
            Key: { id: "123" },
          },
        },
      ],
      TableTwo: [
        {
          PutRequest: {
            Item: {
              id: "234",
              name: "dynobase",
              email: "dynobase@dynobase.dev",
              description: "Professional DynamoDB Client",
            },
          },
        },
      ],
    },
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

If you are curious about performance of batchWrite, head to dynamodb-performance-testing repo by Alex DeBrie.

Query for a Set of Items

If your table has composite key (which is the best practice), in order to get a collection of items sharing the same Parition Key, use Query method. It also allows to use multiple operators for SortKey such as begins_with or mathematical ones like >, =, >= and so on.

const AWS = require('aws-sdk');
const dynamoDB = new AWS.DynamoDB.DocumentClient({region:'us-east-1'});

dynamoDB
  .query({
    TableName: 'my-table',
    KeyConditionExpression: 'id = :hashKey and date > :rangeKey'
    ExpressionAttributeValues: {
      ':hashKey': '123',
      ':rangeKey': 20150101
    }
  })
  .promise()
  .then(data => console.log(data.Items))
  .catch(console.error);

Keep in mind that Query can return up to 1MB of data and you can also use FilterExpressions here to narrow the results on non-key attributes.

If you don't know how to construct your Query and its attributes, head to our DynamoDB Query Builder which will generate code for you. You can also use Query Code Generation feature inside Dynobase.

Simple Transaction

DynamoDB also support transactions - they allow to run multiple write operations atomically meaning that either all of operations are executed succesfully or none of them. It is especially useful when dealing with applications where data integrity is essential, e.g. in e-commerce - adding an item to a cart and decrementing count of items still available to buy.

Such flow should:

Should happen atomically - these two operations should be treated as one, we don't want to have a single moment in time where there's a discrepancy in items count
Should succeed only if count of items available to buy is greated than zero

Described flow can be modelled in Javascript & DocumentClient like this:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region:'us-east-1' })

await dynamoDB.transactWrite({
  TransactItems: [
    {
      Put: { // Add item to cart
        Item: {
          id: '1',
          count: '1'
        }
        TableName: "CartTable",
      },
    },
    {
      Update: { // decrement count of items available to buy only if the count is greater than zero
        ConditionExpression: "#count > 0",
        ExpressionAttributeNames: { "#count": "count" },
        ExpressionAttributeValues: {
          ":value": 1,
        },
        Key: {
          id: '123',
        }
        TableName: "ItemsTable",
        UpdateExpression: "SET #count = :count - :value",
      },
    },
  ],
}).promise();

If you want to learn more about transactions, head to our DynamoDB Transactions Guide.

Read Transaction

Transactions can be also used for reading data atomically. Like in batchGet, you can retrieve the data from multiple tables in a single call:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: "us-east-1" })

await dynamoDB
  .transactGet({
    TransactItems: [
      {
        Get: {
          TableName: "TableOne",
          Key: {
            HashKey: "abcd",
          },
        },
      },
      {
        Get: {
          TableName: "TableTwo",
          Key: {
            HashKey: "1234",
          },
        },
      },
    ],
  })
  .promise()

If you want to learn more about transactions, head to our DynamoDB Transactions Guide.

Query with Sorting

const AWS = require('aws-sdk');
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: 'us-east-1' });

dynamoDB
  .query({
    TableName: 'my-table',
    IndexName: 'Index', // Main one
    KeyConditionExpression: 'id = :hashKey and date > :rangeKey'
    ExpressionAttributeValues: {
      ':hashKey': '123',
      ':rangeKey': 20150101
    },
    ScanIndexForward: true // true or false to sort by "date" Sort/Range key ascending or descending
  })
  .promise()
  .then(data => console.log(data.Items))
  .catch(console.error);

Query (and Scan) DynamoDB Pagination

Both Query and Scan operations return results with up to 1MB of items. If you need to fetch more records, you need to invoke a second call to fetch the next page of results. If LastEvaluatedKey is present in response object, this table has more items like requested and another call with ExclusiveStartKey should be sent to fetch more of them:

const getAll = async () => {
  let result, accumulated, ExclusiveStartKey;

  do {
    result = await DynamoDB.query({
      TableName: argv.table,
      ExclusiveStartKey,
      Limit: 100,
      KeyConditionExpression: 'id = :hashKey and date > :rangeKey'
      ExpressionAttributeValues: {
        ':hashKey': '123',
        ':rangeKey': 20150101
      },
    }).promise();

    ExclusiveStartKey = result.LastEvaluatedKey;
    accumulated = [...accumulated, ...result.Items];
  } while (result.Items.length || result.LastEvaluatedKey);

  return accumulated;
};

getAll()
  .then(console.log)
  .catch(console.error);

Update Item

DynamoDB update operation in Node.js consists of two main parts:

Part which item to update (Key), similar to get
Part what in the selected item should be updated (UpdateExpression and ExpressionAttributeValues)

const AWS = require("aws-sdk")
AWS.config.update({ region: "us-east-1" })
const dynamoDB = new AWS.DynamoDB.DocumentClient()

dynamoDB
  .update({
    TableName: "my-table",
    Key: {
      id: "123",
    },
    UpdateExpression: `set firstName = :firstName`,
    ExpressionAttributeValues: {
      ":firstName": "John McNewname",
    },
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

Conditionally Update Item

Sometimes we want to update our record only if some condition is met, e.g. item is not soft-deleted (does not have deletedAt attribute set). To do that, use ConditionExpression which has similar syntax to the FilterExpression:

const AWS = require("aws-sdk")
AWS.config.update({ region: "us-east-1" })
const dynamoDB = new AWS.DynamoDB.DocumentClient()

dynamoDB
  .update({
    TableName: "my-table",
    Key: {
      id: "123",
    },
    UpdateExpression: `set firstName = :firstName`,
    ExpressionAttributeValues: {
      ":firstName": "John McNewname",
      ":company": "Apple",
    },
    ConditionExpression: `attribute_not_exists(deletedAt) and company = :company`,
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

In this example the name attribute of the record with partition key id = 123 in table my-table will be only updated if this item does not have attribute deletedAt and its attribute company has value Apple.

Increment Item Attribute

Incrementing a Number value in DynamoDB item can be achieved in two ways:

Get item, update the value in the application code and send a put request back to DDB overwriting item
Using update operation

While it might be tempting to use first method because Update syntax is unfriendly, I strongly recommend using second one because of the fact it's much faster (requires only one request) and atomic:

const AWS = require("aws-sdk")
AWS.config.update({ region: "us-east-1" })
const dynamoDB = new AWS.DynamoDB.DocumentClient()

dynamoDB
  .update({
    TableName: "my-table",
    Key: {
      id: "123",
    },
    UpdateExpression: `set score = :score + :value`,
    ExpressionAttributeValues: {
      ":value": 1,
    },
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

In this example the score attribute of the record with partition key id = 123 in table my-table will incremented by one. Of course, you can use other mathematical operators too.

Delete Item

Removing single item from table is very similar to Get Item operation. The parameters of the call are actually exactly the same, the only difference is that we call delete instead of get:

const AWS = require("aws-sdk")
const dynamoDB = new AWS.DynamoDB.DocumentClient({ region: "us-east-1" })

dynamoDB
  .delete({
    TableName: "my-table",
    Key: {
      id: "123", // id is the Partition Key, '123' is the value of it
    },
  })
  .promise()
  .then(data => console.log(data.Attributes))
  .catch(console.error)

Delete All Items

const truncateTable = async () => {
  let ExclusiveStartKey, result

  do {
    result = await DynamoDB.scan({
      TableName: argv.table,
      ExclusiveStartKey,
    }).promise()

    ExclusiveStartKey = result.LastEvaluatedKey

    console.log(`Found ${result.Items.length} Items, removing...`)

    if (result.Items.length > 0) {
      await Promise.all(
        result.Items.map(async item =>
          DynamoDB.delete({
            TableName: argv.table,
            Key: {
              pk: item.pk,
              sk: item.sk,
            },
          }).promise()
        )
      )
    }
  } while (result.Items.length || result.LastEvaluatedKey)
}

truncateTable()
  .then(() => console.log("Done!"))
  .catch(console.error)

If you don't want to run this script on your own (it might be really time consuming), truncating table is possible just with few clicks using Dynobase.

Run DynamoDB Local

const AWS = require("aws-sdk")

const dynamoDB = new AWS.DynamoDB({
  // or DocumentClient
  endpoint: "http://localhost:8000",
})

If you want to see your local tables and data in them, you can use Dynobase to query and modify items in offline tables.

Learn more about running DynamoDB locally.

Bonus!

If you're looking for Serverless Framework based CRUD project which uses other AWS Lambda and DynamoDB best practices with Node.js, you can find one on our Github.

12 Common Misconceptions about AWS DynamoDB

Rafal Wilinski — Sun, 30 May 2021 15:04:06 +0000

It's NoSQL database so it's not suitable for relational data

A common misconception I often hear about DynamoDB is that because it's NoSQL and does not support JOINs like traditional relational databases, it is not suitable for relational data. Well, modeling relationship is perfectly doable in DynamoDB.

he two most common approaches use AWS Amplify, which provides all AWS resources, including tables and resolvers, and a single-table design, which uses only one table to fit all data entities into one container and smart keys composition. While I highly recommend the former to get started and for smaller projects, the latter is more "professional" and officially recommended by AWS. If you want to learn more about single-table design, you can learn about it from Alex's DeBrie book about DynamoDB.

In fact, DynamoDB is suitable for almost all types of data. It makes a perfect key value store, metadata store, relational database, event store (e.g. in Event Sourcing) and transactional data store - thanks to transactions support.

DynamoDB is slow

Anoher argument is about the speed. Many times I've heard that developers can fetch data from their relational databases in less than 1ms! And DynamoDB? Same operation takes 10ms or even 20ms, it's too slow, right?

Not really. These scenarios are often based on an oversimplified setups where the speed is measured by fetching one row by an indexed field and on a beefy machine without traffic, erratic spikes and a myriad of other factors. Reality is often a lot messier, especially at scale. As your relational database starts getting more and more traffic, you'll encounter slowdowns related to the load on the machine caused by other operations and processes, connection pool exhaustions, transaction conflicts, and so on.

What about DynamoDB in such conditions? The performance is always the same. No matter if you're sending 1 request per second, or 1,00,000 requests per second, DynamoDB (if data model has been architected correctly), behaves great, sometimes even better under heavy load.

Source: Amazon DynamoDB auto scaling: Performance and cost optimization at any scale

DynamoDB is expensive

Comparing to traditional, non-managed databases, it's much cheaper to scale. Actually, DynamoDB costs are super predictable and directly proportional to the usage. In traditional, non-managed databases, the TCO (Total Cost of Ownership) is much more non-linear and has a lot of hidden costs that might be not visible at first glance:

At the beginning, on a small scale, DynamoDB costs are close to zero since if there's no traffic, there are no costs. Moreover, AWS offers quite generous Free Tier so you might even go to the production with zero database costs. In a non-managed database, you need to provision a VM/Machine/Instance with at least 1 vCPU. That's a hard cost you cannot skip and you're paying it even if your database is completely unused.

Later on, your provisioned database might outperform DynamoDB in terms of costs but at some point you'll encounter a point where current machine will be not enough. You will find yourself in a constant struggle of bumping the specification, investigating transient and hard to debug problems, or tweaking not-so-well documented variables. At some point, you might even consider hiring a dedicated specialist with a hefty pay to deal with these problems for you.

At the same time, DynamoDB requires zero supervision or fine-tuning. It simply works. And while that sometimes it might seem more expensive, you're actually saving a lot of money (and time!) by not having to worry about operational issues, managing backups, and having constant throughput and 99.99% SLA.

Remember, always compare TCO, not just pure cost of running VMs/machines/containers.

DynamoDB is schemaless

That's actually not a misconception. Although DynamoDB is giving users freedom of items shape - they don't have to conform to any schema since DynamoDB is schemaless, lack of any validation or convention might lead to a huge mess.

However, if you're going to design your data model according to single-table design, you're going to have a really strict set of usable access patterns which essentially will become your schema.

DynamoDB is hard to use

Every technology requires some investment and DynamoDB is no exception. However, because DynamoDB API is very minimal (DocumentClient has only 11 operations), the scope of the material to become an expert might be a lot smaller compared to learning PostgreSQL or Elasticsearch. Furthermore, there are books about just running and maintaining these two databases. In DynamoDB that problem simply does not exist. It's ran by Amazon for you and you don't need to worry about it.

You might also argue that lack of SQL support is a blocker. Up until recently, without support for PartiQL, this statement was true but right now, you can use it. However, if you want to use native query language, we've prepared a visual query builder which might help you getting started.

Moreover, the ecosystem of DynamoDB is constantly growing. There are more and more tools providing useful abstractions like DynamoDB Toolbox by Jeremy Daly for working with single-table design, Dynamoose for ORM-like experience, AWS Amplify which hides the DynamoDB layer from the programmer completely, or Dynobase which helps you in navigating between profiles, regions, tables and exploring datasets.

Lastly, DynamoDB integrates with the rest of the AWS ecosystem seamlessly reducing the amount of code you'd need to create. Mechanisms like TTL, Point in Time recovery, streaming, global replication are built into it. Your responsibility is just to use them, not author them.

Cannot use SQL with DynamoDB

This statement was true until some time ago. Recently, AWS announced PartiQL support for DynamoDB. With PartiQL, an SQL-like language, you can use familiar language to interact with DynamoDB and other AWS' components like Athena.

Keep in mind that using SQL is not removing DynamoDB's technical limitations. You still need to understand that your SELECT * FROM ... queries will be translated to scans or queries, that things like UPDATE X WHERE Y probably is not going to work, or that COUNTing is also not permitted.

Only suitable for Serverless workloads

DynamoDB is very often being composed into Serverless-based architectures. In fact, it is a perfect match because just like AWS Lambda, S3, and other managed services, it is billed for the actual usage of it - per amount of read / write operations and gigabytes of disk space your tables consume.

That being said, it is not blocking you from using DynamoDB with non-Serverless workloads. Tables can be accessed from any environment, including EC2 instances, CI/CD systems, VMs, containers, on-premises or even your local machine. All you need to is to have a valid AWS identity with credentials scoped to allow accessing it. With that in place, you can use CLI or SDKs to interact with DynamoDB.

DynamoDB is hard to manage

Totally untrue. No, it's all managed for you. Once you click "Create Table" in the AWS Console, you get a highly available, redundant, scalable, encrypted both at rest and in transit, SOC, PCI and HIPAA compliant database within seconds. Imagine how hard that would be to achieve the same result using on-premise software.

But, wait there's more. With DynamoDB, you can enable cross-region replication with sub-second latency, start streaming data from it to other data sources or data processors, or enable PITR (Point in Time Recovery) with a single API call. How cool is that? And once again, how much harder that would be using non-managed software?

Essentially, DynamoDB allows you to build product on a shoulders of giants allowing you to focus on what's important for your business, on your core competency. Friends don't let friends run databases on instances/VMs/containers/on-premises.

DynamoDB is secure by default

Just like any other database or service, DynamoDB is just a tool. While it has been built with the best security principles in mind, including encryption, IAM, working backups, it's up to you how you're going to use it. Shared responsibility model is key here.

While Amazon ensures the best "security of the cloud", you're still responsible for the "security in the cloud". No one is going to prevent you from storing passwords in plaintext, using wildcard IAM policies, leaking your AWS identity credentials or violating laws and regulations.

DynamoDB actually makes the security of your database easier. Because there are no servers, containers, VMs or clusters to manage, you don't have to worry about restricting network access correctly, patching software regularily, or being up-to-date with latest vulerabilities.

DynamoDB cannot be used for serious projects

That's simply not true. There's a number of really serious companies using DynamoDB in production including Duolingo, an online learning site, uses DynamoDB to store approximately 31 billion data objects on their web server, Nike, who ditched their Cassandra cluster in favor of DynamoDB, or Disney that uses DynamoDB to store metadata of billions of customer actions.

DynamoDB is also dogfood by Amazon internally - it is a Tier-0 service powering most of Amazon. It is also highly advised for all new initiatives at Amazon to use DynamoDB as the database.

DynamoDB is suitable only for big projects with huge traffic

False! Thanks to free tier, you can start using DynamoDB completely for free. After that, the usage costs are scaling as you go. The more clients you get, the more money you earn, the bigger DynamoDB costs are and everything is proportional. Moreover, there's no initial commitment like in other non-managed databases where the entry instance starts with 10USD / month.

DynamoDB is bad for analytics

The last point is a bit more complicated. It is true that because you cannot run arbitrary queries against DynamoDB tables (technically speaking you can but it's extremely inefficient due to Scan nature), you cannot perform ad-hoc reports of your data. That might be disappointing. However, there are solutions to that:

You can stream your data from DynamoDB to Redshift or some other relational database built for reporting and analytics purposes
You can build aggregation functions on your own. These functions plugged to the DynamoDB Streams will be automatically recalculating your composite statistics or rollups providing always the most up-to-date state of some aggregate.

10 Tips to Optimize Your DynamoDB Costs

Rafal Wilinski — Sun, 30 May 2021 14:34:02 +0000

Because Amazon DynamoDB is a fully managed database where a user does not provision any machines, the pricing is not as straight forward. While our Pricing Calculator can help with forecasting the costs, the pricing optimization techniques are a bit different. Below are 10 tips that can help you lower your DynamoDB costs:

1. Use cheaper regions

If you are not concerned about your data's location because you don't need to meet any regulations or compliance standards, you can provision your tables in regions where it's cheaper. For example, if your table will have 100GB of data, the storage will cost $28.50 per month in Tokyo (ap-northeast-1), $29.72 per month in London (eu-west-2), or even $37.50 per month in Sao Paulo (sa-east-1).

The cheapest regions are us-east-1, us-east-2 and us-west-2 costing $0.25 per GB/month, $0.00065 per WCU/hour and $0.00013 per RCU/hour.

2. Use shorter attribute names

Because DynamoDB in both On-Demand and Provisioned capacity mode uses size-dependent billing units (1 WCU/WRU = 1KB, 1 RCU/RRU = 4KB), plus, you're paying for storage too, you should always aim to make your records as small as possible. If making attribute values is not an option, try making attribute names shorter. This helps you reduce the amount of storage required for your data. Moreover, when storing dates, you can choose the epoch time format instead of ISO dates because it's shorter too.

3. Be aware of huge blobs

Saving images in DynamoDB can quickly skyrocket costs. It's very inefficient, and you should rather store all images or linked assets in S3 and save the URL pointing to it in DDB. If that's not an option, consider using compression algorithms like gzip to make blobs smaller before saving them.

4. Prefer queries over scans

DynamoDB has two ways of fetching a set of records from it: Query and Scan. While the query is using partition and sort key to get the desired piece of data fast and directly, the scan, on the other hand, is "scanning" through your whole table. The difference here is that while in Query, you are charged only for items which are returned, in scan case, you're being charged for all the rows scanned, not the total amount of items returned.

5. Avoid strongly consistent reads and transactions where possible

DynamoDB uses eventually consistent data model. It means that updates are propagated across all storage locations, usually within one second or less. However, DynamoDB supports strongly consistent reads too, but with an additional cost. Strongly consistent reads require a double amount of Read request/capacity units than the eventually consistent reads.

6. When using GSIs, think about Attribute Projections

Attribute Projections specify which attributes are available when querying for data using Global Secondary Index. Sometimes, when accessing the data using GSIs, not all attributes are needed. Reducing the amount of data available in GSIs by using Attribute Projection KEYS_ONLY or INCLUDES instead of ALL will reduce the amount of data kept in GSI significantly thereby lowering not only the costs storage but also consume less read/write units when accessing or updating the data.

7. Use on-demand mode wisely

As Yan Cui demonstrated in his blog post, "rough estimate suggests that on-demand tables are around 5–6 times more expensive per request compared to a provisioned table". Conclusion: if your workload has steady utilization without sudden spikes, consider going with provisioned mode with autoscaling enabled. You can simulate costs by using our DynamoDB Pricing Calculator.

8. Use reserved capacity

When using provisioned capacity mode and your capacity is bigger than 100 units, you can also consider purchasing reserved capacity. For a three year term, reserved capacity provides a 76% discount, and for a one year term, reserved capacity provides a 53% discount when compared to provisioned throughput capacity.

9. Remove unnecessary items

To minimize the storage costs, aim to always purge unnecessary data. If your business does not have to keep the events older than X days, you can leverage the TTL functionality.

Moreover, if you have any many-to-many relationships, remember that you need to take care of "orphaned records" on your own. As DynamoDB is not a relational database, it does not have an ON DELETE CASCADE feature like many RDBMS.

10. Do not overuse GSIs

DynamoDB automatically copies the right set of attributes to any indices where the attributes must exist. This increases the costs of your storage because of the additional space used. In detail, it is a sum of:

Byte size of table primary key
Byte size of index key attribute
Byte size of projected attributes
100 byte-overhead per index item

Show Dev.to: I made an alternative UI for DynamoDB admin, check it out!

Rafal Wilinski — Wed, 12 Feb 2020 09:33:28 +0000

👋 Hi Dev.to Community!

Around 6 months ago, I spent 3 days debugging some data problems in DynamoDB. While it’s a great database, I found the experience of the console a bit lacking. Jumping from Chrome window to Chrome window, I kept constantly retyping the same queries, the pagination was annoying, and the speed wasn’t right. I wanted to change that.

Finally, I’m super happy to announce that Dynobase is live after 5 months of development. This is a result of my own pains working with DynamoDB console and community feedback.

It’s free to use for 7 days.

Some of the notable features are:

⚙️ Automatic Query Code and Types generation
🔖 Bookmarks and History
👨‍👩‍👧‍👦 Multiple profiles, tables, and tabs support
📥 Import/Export from and to CSV/JSON
🚨 DynamoDB Offline support

I would love to hear what you think, comment below and let me know your feedback / questions, thanks!

Download link: https://dynobase.dev/

Creating AWS Amplify Functions in Typescript

Rafal Wilinski — Tue, 15 Oct 2019 13:24:10 +0000

AWS Amplify gives a great power to the developers. With just a few commands you can unlock true power of Cloud and provision whole APIs, add some Lambda functions and host in on S3 with CloudFront CDN in front of it. However, right now Amplify supports creating only Javascript-powered functions. In this article I'll explain how you can create your functions in Typescript. Moreover, following method should work for other languages such as Kotlin, ReasonML or ClojureScript!

Let's start with creating Amplify project and custom function

Create base project using React or any other boilerplate generator.

npx create-react-app amplify-typescript-functions

Initialise Amplify project

amplify init

and proceed through Amplify project creation wizard...

After you created project, go ahead and add a function with following command

amplify function add

And once again proceed through guided creation process.

After that you should see a new folder amplify/backend/function/<YOUR FUNCTION NAME> appear in your project structure.

> amplify-typescript-demo/amplify/backend $ tree
.
├── amplify-meta.json
├── awscloudformation
│   └── nested-cloudformation-stack.yml
├── backend-config.json
└── function
    └── amplifytypescriptdemf00d9d47
        ├── amplifytypescriptdemf00d9d47-cloudformation-template.json
        ├── function-parameters.json
        └── src
            ├── event.json
            ├── index.js
            └── package.json

Adding Typescript function

We have our desired function but it's in Javascript and we want to add some types to it. So, go ahead and remove index.js and replace it with index.ts. Paste following code inside:

import { APIGatewayProxyHandler } from 'aws-lambda';
import 'source-map-support/register';

export const handler: APIGatewayProxyHandler = async (event,  _context) => {
  console.log(event);
  return  {
    statusCode:  200,
    body: JSON.stringify({
      message:  'Amplify function in Typescript!',
      input:  event,
    }),
  };
};

Typescript also need some instructions what to do with following code and how to compile it so we'll need tsconfig.json file too. Here's what should be inside:

{
  "compilerOptions":  {
    "lib":  ["es2017"],
    "moduleResolution":  "node",
    "noUnusedLocals":  true,
    "noUnusedParameters":  true,
    "sourceMap":  true,
    "target":  "es2017",
    "outDir":  "lib"
  },
  "exclude":  ["node_modules"]
}

package.json will also need some adjustments:

"dependencies":  {
  "source-map-support": "0.5.10"
},
"devDependencies":  {
  "@types/aws-lambda": "8.10.17",
  "@types/node": "10.12.18"
}

You might also consider adding .gitignore file add exclude all *.js files there.

Putting it all together

Lastly, we need to install typescript as devDependency at the root of our project and tell Amplify what to do with our TS-based function. Fortunately, recently Amplify added Build Options for Functions. With that in mind, we can add following command to package.json:

"scripts":  {
  "amplify:<YOUR_FN_NAME>":  "cd amplify/backend/function/<YOUR_FN_NAME>/src && yarn && tsc ./*.ts"
}

With this little piece, once you execute amplify push following script will be executed and build TS source files before deploying anything.

You can test it locally using amplify function invoke <YOUR_FN_NAME> (make sure it's built first!). Final result is also available as Github Repository here

Following example works really well if number of functions is quite low (because each function converts directly to extra command in package.json and there is no shared code between them. In the upcoming article I'll explore different ways to do that more efficiently, stay tuned!

If your project uses DynamoDB as a data store, check out my cloud startup - Dynobase - Professional GUI Client for DynamoDB
. It should make your life a bit easier.

Hybrid Load Balancing traffic between EC2 and Lambda functions - useful API migration pattern

Rafal Wilinski — Sat, 24 Aug 2019 09:51:47 +0000

With this particular PR included in version 1.45, the Serverless Framework team has finally enabled a possibility of using Application Load Balancers as event source instead of using API Gateway.

We've heard few of stories where replacing costful API Gateway with ALB has saved thousands of dollars and it's really awesome but here I'd like to talk about another aspect which might be really helpful in migrations:

Hybrid Load Balancing

Imagine following scenario: your company wants to migrate from ECS/EC2 to Lambda by stripping monolith web server route by route. It appears that such thing can be achieved easily with multiple target groups associated with Load Balancer. Instead of just one, you can specify many of them and basing on request paths (or whatever you like) route requests to EC2/ECS or to Lambda.

How to do that?

If you don't have existing infra, provision VPC, EC2 and ALB, prefereably in separate files to keep it simple and clean. Otherwise, use existing ALB ARN.
Reference supplement CloudFormation files in main serverless.yml file:

resources:
  - ${file(./infra/alb.yml)}
  - ${file(./infra/ec2.yml)}
  - ${file(./infra/vpc.yml)}

Replace traditional http event with alb where you specify which Load Balancer, Condition and Priority. In this case all requests to path /lambda will be routed to Serverless solution. Rest of them will be routed to the EC2 based on rule specified here

functions:
  hello:
    handler: handler.handle
    events:
      - alb:
          listenerArn:
            Ref: albListener
          priority: 1
          conditions:
            path: /lambda

One thing worth noting here is that Lambda priority should be bigger than server's, so it should look something like this:

If you want to check fully working example, the source code can be found here

Just run it with:

serverless deploy

This article was originally published at servicefull.cloud

Bulletproof ECS Cluster - Best practices for creating secure, stable and cost-effective clusters on AWS

Rafal Wilinski — Tue, 30 Jul 2019 07:03:39 +0000

Running ECS Cluster on AWS may be troublesome. After bunch of outages and war stories, I present you a list of lessons learned the hard way to make your cluster secure and anti-fragile on a budget.

Host in private, expose as via ALB/ELB

The Load Balancer should be your gateway to the cluster. When running a web service, make sure you've running your cluster in a private subnet and your containers cannot be accessed directly from the internet. Ideally, your internal container should expose a random, ephemeral port which is bound to a target group. Make sure also that traffic is only allowed from the Load Balancer's Security Group.

Use Spot Instances

Spot Instances are a great way to save money on Compute Heavy/Worker clusters or for dev/staging environments. If you don't know what spot instances are, here's a quote from AWS Docs:

Spot Instance is an unused EC2 instance that is available for less than the On-Demand price.

In reality, by using them you can save up to 80% of costs! But, there's a catch. Spot Instances are based on bid pricing model and AWS can kill them spontaneously so, you should be aware that your instances might be taken down anytime with one minute warning.

Use Parameter Store for environment variables

In container's task definition you can specify environment variables directly, however, it's not the best way to pass config variables to the container. It's not secure, it couples infrastructure with env variables, changing them requires you to change task definition and perform a new deployment. Instead, environment variables should be rather injected during start from AWS SSM (Simple Systems Manager) Parameter Store.

It's a secure, encrypted, auditable, scalable and managed central source of truth which might be used not only by ECS but also by Lambda, EC2, CloudFormation and many more. Moreover, it integrates with other AWS services nicely, so once a variable changes, you might react on that event. More on that on official AWS blog.

Update ECS Agent on EC2 instance start

> service MY-SERVICE was unable to place a task because no container instance met all of its requirements.

That's the error we saw one day when investigating an outage. "What's wrong?" - we kept asking that question to ourselves. After digging for a while, we found a root cause.

Because we were running our cluster on Spot instances, AWS decided to terminate them. Luckily, our Autoscaling Group re-spawned them to maintain a desired amount of running instances but there was one problem. The AMI had outdated ECS agent installed and it had no capability to pull SSM parameters from the store, so the task refused to start repeatedly.

Solution? Include these two simple lines in your EC2 Auto Scaling Group user data ensuring ECS agent is up-to-date on launched instances:

sudo yum update -y ecs-init

# Depending on AMI
### On ECS-optimized Amazon Linux 2 AMI
sudo systemctl restart docker

### On ECS-optimized Amazon Linux AMI
sudo service docker restart && sudo start ecs

Scale instances based on reservations, not usage

Remember service MY-SERVICE was unable to place a task... error? The other case for that might be just the fact that your cluster has not enough resources to place it. It's a very common scenario when CPU reservations go up to 80% but usage stays at the very low level preventing from scale-up operation.

Typical scaling operation problem. Usage very low but reservations quite high.

That's why you want to scale you EC2 machines running your cluster based on Memory or CPU reservations instead of usage. Your services should scale based on usage and as the amount of them grows, the amount of reserved resources also, which will effectively trigger AutoScaling group operation on underlying EC2s.

Adjust healthcheck grace period and scaling cooldowns

Knowing approximately how long does it take for your container to start is a crucial information to adjust healthcheck grace period correctly. If you configure it too aggressively, your instances which are still launching might be prematurely marked as unhealthy and killed by scheduler before they even manage to start. This will trigger a never-ending cycle of instances being provisioned and killed. In most cases, increasing this period should solve your problem.

On the other hand, making grace period too long might lead to slow deployments and make your cluster not scale well. My advice is to start from 60 seconds and check whether that amount is enough.

Use Target Track Scaling instead of Step Scaling

Unlike step scaling where you set up a two thresholds for scale up or down operations, in target track scaling service automatically calculates the amount of services/instances to keep the metric at, or close to specified value. This is not only easier to set up as requires much less configuration but also more effective from my experience.

Or maybe even try custom scaling strategy with Events inside Lambda

If autoscaling options provided by AWS are not sufficient for your needs, you can always include custom scaling logic inside Lambda function which reacts to ECS Events, Metrics and Alarms. You can find more on that here.

Go multi-AZ

AZ stands for Availability Zone, you can think of AZ as an independent data center. In history, we had some cases when whole AWS AZs went down so it's likely to happen in the future. However, it's quite simple to protect your cluster from similar outage - just distribute your machines in multiple AZs.

While Auto Scaling group manages the availability of the cluster across multiple AZs to give you resiliency and dependability, the ECS scheduler manages task distribution across these underlying machines in different zones effectively making your cluster highly available and independent of single AZ failures.

Configure blue-green deployment

One challenge of achieving full agility in engineering is allowing engineers to make deployments a breeze, rather than a task requiring downtime and whole team keeping a finger on a pulse. That's where Blue-Green Deployment comes to play.

When performing a deployment, exact same copy of current environment/service is spawned and the traffic gradually becomes routed to the new target, increasing by few percent each minute eventually reaching 100% on a new unit. Once all the traffic goes to the new target and it's stable, green unit becomes decommissioned and the new version aka "blue" becomes the new standard aka "green". When new version is malfunctioning by for example responding with too big amount of 400s, the procedure is stopped and traffic gets rolled back to green fully.

This method allows your dev team to push new version of the code few times a day without worrying about downtime and pushing faulty code to the production. You can read more about it on official AWS blog.

Use IaaC

Tools like Terraform, Cloudformation or Cloud Development Kit (CDK) are a perfect way provision your cluster in reproducible way. This is especially helpful when your client requests from you an additional environment or you need to run dynamic, per-branch staging environments. Moreover, you can find many plug'n'play, really well made modules according to these best practices will which create such cluster with a single click.

Use CloudWatch logs

To make troubleshooting of your applications much easier, always push containers STDOUT logs to CloudWatch. You can do that simply by adding following lines to containerDefinitions:

"logConfiguration": {
  "logDriver": "awslogs",
  "options": {
    "awslogs-group": "awslogs",
    "awslogs-region": "us-west-2",
    "awslogs-stream-prefix": "awslogs-example"
  }
}

Don't forget about attaching necessary IAM permissions: logs:CreateLogGroup, logs:CreateLogStream, logs:PutLogEvents, logs:DescribeLogStreams

As soon as they are in CloudWatch, you can very easily find relevant information using Insights.

Also, in your application code remember to change log line endings from \n to \r to save multiline strings as one entry. This will not only save your money but also make them greppable.

If you liked this post, check out my cloud startup - Dynobase - Professional GUI Client for DynamoDB