Forem: Joe Stech

Automated Cloud Migrations with Kiro and the Arm MCP Server

Joe Stech — Sat, 20 Dec 2025 00:11:52 +0000

AWS Graviton has the best price-performance of any EC2 instance on AWS, and with the recent announcement of Graviton5, the value prop just keeps getting better and better. Migrating to Graviton makes both financial sense and will provide a huge performance boost to your applications. Most of the time your applications will just move over seamlessly, but what if you've got some x86-specific optimizations in your code?

Great news: you don't have to manually migrate anymore.

In this post, I'll show you how to use Kiro (AWS's agentic IDE) combined with the Arm MCP Server to automate the entire migration process. We're talking Docker images, SIMD intrinsics, compiler flags -- the whole thing.

What's the Arm MCP Server?

The Arm MCP Server implements the Model Context Protocol, which is basically a way for AI coding assistants to tap into specialized tools. When you connect it to Kiro, you suddenly have an agent that can:

Check Docker images for arm64 support without you having to dig through manifests
Scan your codebase for x86-specific code (intrinsics, build flags, etc.)
Search Arm's knowledge base for migration guidance and intrinsic equivalents
Analyze assembly for performance characteristics

The Problem: A Legacy x86 Application

Let's walk through an example of how you might use the Arm MCP Server with Kiro. Pretend you've inherited a legacy benchmarking application that's deeply tied to x86. Here's the Dockerfile:

FROM centos:6

# CentOS 6 reached EOL, need to use vault mirrors
RUN sed -i 's|^mirrorlist=|#mirrorlist=|g' /etc/yum.repos.d/CentOS-Base.repo && \
    sed -i 's|^#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-Base.repo

# Install EPEL repository (required for some development tools)
RUN yum install -y epel-release && \
    sed -i 's|^mirrorlist=|#mirrorlist=|g' /etc/yum.repos.d/epel.repo && \
    sed -i 's|^#baseurl=http://download.fedoraproject.org/pub/epel|baseurl=http://archives.fedoraproject.org/pub/archive/epel|g' /etc/yum.repos.d/epel.repo

# Install Developer Toolset 2 for better C++11 support (GCC 4.8)
RUN yum install -y centos-release-scl && \
    sed -i 's|^mirrorlist=|#mirrorlist=|g' /etc/yum.repos.d/CentOS-SCLo-scl.repo && \
    sed -i 's|^mirrorlist=|#mirrorlist=|g' /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo && \
    sed -i 's|^# baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-SCLo-scl.repo && \
    sed -i 's|^# baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo

# Install build tools
RUN yum install -y \
    devtoolset-2-gcc \
    devtoolset-2-gcc-c++ \
    devtoolset-2-binutils \
    make \
    && yum clean all

WORKDIR /app
COPY *.h *.cpp ./

# AVX2 intrinsics are used in the code
RUN scl enable devtoolset-2 "g++ -O2 -mavx2 -o benchmark \
    main.cpp \
    matrix_operations.cpp \
    -std=c++11"

CMD ["./benchmark"]

This will cause problems when trying to migrate to Graviton:

centos:6 doesn't support arm64
-mavx2 is an x86-only compiler flag
The code uses AVX2 intrinsics (spoiler: won't compile on Arm)

If you didn't spot these issues yourself, that's fine! Kiro with the Arm MCP Server will.

Here's what that matrix multiplication code looks like:

#include "matrix_operations.h"
#include <iostream>
#include <random>
#include <chrono>
#include <stdexcept>
#include <immintrin.h>  // AVX2 intrinsics

Matrix::Matrix(size_t r, size_t c) : rows(r), cols(c) {
    data.resize(rows, std::vector<double>(cols, 0.0));
}

void Matrix::randomize() {
    std::random_device rd;
    std::mt19937 gen(rd());
    std::uniform_real_distribution<> dis(0.0, 10.0);

    for (size_t i = 0; i < rows; i++) {
        for (size_t j = 0; j < cols; j++) {
            data[i][j] = dis(gen);
        }
    }
}

Matrix Matrix::multiply(const Matrix& other) const {
    if (cols != other.rows) {
        throw std::runtime_error("Invalid matrix dimensions for multiplication");
    }

    Matrix result(rows, other.cols);

    // x86-64 optimized using AVX2 for double-precision
    for (size_t i = 0; i < rows; i++) {
        for (size_t j = 0; j < other.cols; j++) {
            __m256d sum_vec = _mm256_setzero_pd();
            size_t k = 0;

            // Process 4 elements at a time with AVX2
            for (; k + 3 < cols; k += 4) {
                __m256d a_vec = _mm256_loadu_pd(&data[i][k]);
                __m256d b_vec = _mm256_set_pd(
                    other.data[k+3][j],
                    other.data[k+2][j],
                    other.data[k+1][j],
                    other.data[k][j]
                );
                sum_vec = _mm256_add_pd(sum_vec, _mm256_mul_pd(a_vec, b_vec));
            }

            // Horizontal add using AVX
            __m128d sum_high = _mm256_extractf128_pd(sum_vec, 1);
            __m128d sum_low = _mm256_castpd256_pd128(sum_vec);
            __m128d sum_128 = _mm_add_pd(sum_low, sum_high);

            double sum_arr[2];
            _mm_storeu_pd(sum_arr, sum_128);
            double sum = sum_arr[0] + sum_arr[1];

            // Handle remaining elements
            for (; k < cols; k++) {
                sum += data[i][k] * other.data[k][j];
            }

            result.data[i][j] = sum;
        }
    }

    return result;
}

double Matrix::sum() const {
    double total = 0.0;
    for (size_t i = 0; i < rows; i++) {
        for (size_t j = 0; j < cols; j++) {
            total += data[i][j];
        }
    }
    return total;
}

void benchmark_matrix_ops() {
    std::cout << "\n=== Matrix Multiplication Benchmark ===" << std::endl;

    const size_t size = 200;
    Matrix a(size, size);
    Matrix b(size, size);

    a.randomize();
    b.randomize();

    auto start = std::chrono::high_resolution_clock::now();
    Matrix c = a.multiply(b);
    auto end = std::chrono::high_resolution_clock::now();

    auto duration = std::chrono::duration_cast<std::chrono::milliseconds>(end - start);

    std::cout << "Matrix size: " << size << "x" << size << std::endl;
    std::cout << "Time: " << duration.count() << " ms" << std::endl;
    std::cout << "Result sum: " << c.sum() << std::endl;
}

Here's the header file matrix_operations.h:

#ifndef MATRIX_OPERATIONS_H
#define MATRIX_OPERATIONS_H

#include <vector>
#include <cstddef>

class Matrix {
private:
    std::vector<std::vector<double>> data;
    size_t rows;
    size_t cols;

public:
    Matrix(size_t r, size_t c);
    void randomize();
    Matrix multiply(const Matrix& other) const;
    double sum() const;

    size_t getRows() const { return rows; }
    size_t getCols() const { return cols; }
};

void benchmark_matrix_ops();

#endif // MATRIX_OPERATIONS_H

And main.cpp:

#include "matrix_operations.h"
#include <iostream>

int main() {
    std::cout << "x86-64 AVX2 Matrix Operations Benchmark" << std::endl;
    std::cout << "========================================" << std::endl;

#if defined(__x86_64__) || defined(_M_X64)
    std::cout << "Running on x86-64 architecture with AVX2 optimizations" << std::endl;
#else
    #error "This code requires x86-64 architecture with AVX2 support"
#endif

    benchmark_matrix_ops();

    return 0;
}

That's a lot of x86-specific intrinsic code! But you don't have to convert it manually, Kiro + Arm can do it for you.

Setting Up Kiro with the Arm MCP Server

First, you need to configure Kiro to connect to the Arm MCP Server. Kiro uses JSON configuration files for MCP servers.

Create .kiro/settings/mcp.json in your project root:

{
  "mcpServers": {
    "arm-mcp": {
      "command": "docker",
      "args": [
        "run", "-i", "--rm",
        "-v", "/path/to/your/code:/workspace",
        "armlimited/arm-mcp:1.0.1"
      ]
    }
  }
}

A few things to note:

The command runs the Arm MCP Server via Docker
-v "/path/to/your/code:/workspace" mounts your current project directory so the scanner can access your code. Replace /path/to/your/code with your actual path.

Save the file and Kiro will automatically pick up the new server. You can verify it's connected by typing /mcp in the chat—you should see arm-mcp listed with its tools.

Once that's done, you can do quick checks right in the chat. Try something like:

Check the base image in the Dockerfile for Arm compatibility

Kiro will use the Arm MCP tools and tell you that centos:6 only supports amd64. Cool, but we want to automate the whole thing.

Full Automation With Kiro Steering Documents

To fully automate migrations, you can use Kiro "steering documents" -- markdown files that give the AI persistent context and instructions. Instead of explaining what you want every time, you write it once and reference it.

Create a file at .kiro/steering/arm-migration.md:

---
inclusion: manual
---

Your goal is to migrate a codebase from x86 to Arm. Use the MCP server tools to help you with this. Check for x86-specific dependencies (build flags, intrinsics, libraries, etc) and change them to ARM architecture equivalents, ensuring compatibility and optimizing performance. Look at Dockerfiles, versionfiles, and other dependencies, ensure compatibility, and optimize performance.

Steps to follow:
* Look in all Dockerfiles and use the check_image and/or skopeo tools to verify ARM compatibility, changing the base image if necessary.
* Look at the packages installed by the Dockerfile and send each package to the knowledge_base_search tool to check each package for ARM compatibility. If a package is not compatible, change it to a compatible version. When invoking the tool, explicitly ask "Is [package] compatible with ARM architecture?" where [package] is the name of the package.
* Look at the contents of any requirements.txt files line-by-line and send each line to the knowledge_base_search tool to check each package for ARM compatibility. If a package is not compatible, change it to a compatible version.
* Look at the codebase that you have access to, and determine what the language used is.
* Run the migrate_ease_scan tool on the codebase, using the appropriate language scanner based on what language the codebase uses, and apply the suggested changes.
* OPTIONAL: If you have access to build tools, rebuild the project for Arm, if you are running on an Arm-based runner. Fix any compilation errors.
* OPTIONAL: If you have access to any benchmarks or integration tests for the codebase, run these and report the timing improvements to the user.

Pitfalls to avoid:

* Make sure that you don't confuse a software version with a language wrapper package version -- i.e. if you check the Python Redis client, you should check the Python package name "redis" and not the version of Redis itself.
* NEON lane indices must be compile-time constants, not variables.

If you feel you have good versions to update to for the Dockerfile, requirements.txt, etc. immediately change the files, no need to ask for confirmation.

Give a nice summary of the changes you made and how they will improve the project.

The inclusion: manual bit means this steering doc only kicks in when you reference it. You can also use inclusion: always if you want it active all the time.

Running the Migration

Now for the fun part. In Kiro's chat, just type:

#arm-migration

That's it. Kiro will:

Find your Dockerfile and check centos:6 for arm64 support (it'll fail)
Suggest baseimage replacements
Scan your C++ code with migrate_ease_scan
Find all those AVX2 intrinsics and convert them to NEON
Update the compiler flags
Give you a summary of everything it changed

Verify It Works

After accepting the changes, build and test on an Arm system:

g++ -O2 -o benchmark matrix_operations.cpp main.cpp -std=c++11
./benchmark

You should see something like:

ARM-Optimized Matrix Operations Benchmark
==========================================
Running on ARM64 architecture with NEON optimizations

=== Matrix Multiplication Benchmark ===
Matrix size: 200x200
Time: 12 ms
Result sum: 2.01203e+08

If something breaks, just paste the error back into Kiro and it'll fix it. That's the beauty of agentic workflows -- you're not debugging alone.

Happy migrating! If you run into problems or have questions, you can always email mcpserver@arm.com for help.

Dyson Swarm: How I built a hard science fiction game with AWS services

Joe Stech — Thu, 23 Jan 2025 05:16:33 +0000

The game concept

I'm a huge science fiction fan -- so much so that I started and ran a science fiction magazine for five years, and I'm still very engaged in the science fiction community.

Back in October I had the idea to start making a series of short games to explain hard science fiction concepts in a fun way. Right after I started the game, I saw that AWS was having a Game Builder Challenge, so I decided to write my first hard science fiction game as a part of the hackathon!

Dyson Swarm is an incremental (clicker) game about dismantling the solar system in order to completely envelop the sun in space habitats. You start as a simple game developer, and progress through stages, building up resources as you go along.

I originally thought that development would only take 10-20 hours, but the scope of the game ballooned, and ultimately I ended up spending closer to 70 hours developing the game (the entirety of which were spent after my wife and kids went to sleep at night, which is my excuse if you encounter a bug). Effort estimates are one of the most difficult parts of software engineering!

I've hosted the game here if you want to try it out: Dyson Swarm

The AWS Architecture

The game is built using client-side Javascript, and runs entirely in the browser. Since all the game code is served through static HTML/Javascript elements, the cheapest and easiest way to host such a game is with a static site S3 bucket fronted by the CloudFront CDN -- this allows fast serving of the files all around the world, and CloudFront can do the TLS termination so that you don't need to deal with custom certificates or servers.

The game also collects some high-level metrics about how many people are playing the game (anonymized), and how far they're getting (what stage of the game they've completed). These metrics are stored in an RDS Postgres database, and get there via a serverless API built using API Gateway and Lambda. I'm using an RDS instance, since I already had one available, but you could just as easily use serverless RDS for this.

I deployed the S3 bucket and CloudFront distribution through the AWS console, and created a Python-based AWS CDK stack for the serverless metrics lambda.

The CDK stack

This is the CDK stack for the serverless metrics lambda:

from aws_cdk import (
aws_lambda as lambda_,
aws_apigateway as apigw,
aws_ecr as ecr,
aws_certificatemanager as acm,
aws_route53 as route53,
Duration,
Stack)
from constructs import Construct
import os

class DysonSwarmStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        repo = ecr.Repository.from_repository_name(self, "dysonSwarmRepo", "gamesapi")

        dyson_swarm_lambda = lambda_.DockerImageFunction(self,
            "dysonSwarmLambda",
            code=lambda_.DockerImageCode.from_ecr(
                repository=repo,
                tag=os.environ["CDK_DOCKER_TAG"]
            ),
            memory_size=256,
            timeout=Duration.seconds(60),
            architecture=lambda_.Architecture.ARM_64
        )

        # do auth inside lambda
        api = apigw.LambdaRestApi(self,
            "dysonSwarm-endpoint",
            handler=dyson_swarm_lambda,
            default_cors_preflight_options=apigw.CorsOptions(allow_origins=["*"])
        )

        custom_domain = apigw.DomainName(
            self,
            "custom-domain",
            domain_name="gameapi.compellingsciencefiction.com",
            certificate=acm.Certificate.from_certificate_arn(self,'cert',"[cert ARN here]"),
            endpoint_type=apigw.EndpointType.EDGE
        )

        apigw.BasePathMapping(
            self,
            "base-path-mapping",
            domain_name=custom_domain,
            rest_api=api
        )

        hosted_zone = route53.HostedZone.from_hosted_zone_attributes(
            self,
            "hosted-zone",
            hosted_zone_id="[zone id here]",
            zone_name="compellingsciencefiction.com"
        )

        route53.CnameRecord(
            self,
            "cname",
            zone=hosted_zone,
            record_name="gameapi",
            domain_name=custom_domain.domain_name_alias_domain_name
        )

It's not too complex at all! You can see that it references an existing ECR repo (with a container image for the lambda) along with an existing Route 53 hosted zone (for the custom domain). Other than that, it's creating an API Gateway backed by a Lambda that is built using the specified container in ECR.

This file is also located in the open-source game repo here: dyson_swarm_stack.py if you'd like to see it in context.

The Code

The entire source code for the game can be found here: Dyson Swarm GitHub Repo

The main game loop is a hundred millisecond interval specified at the end of the dysonswarm.html file, so if you want to dig into the code in detail I'd start there.

The game uses local browser storage to maintain state, so if you close the browser and come back later your game is automatically saved. This is actually very easy to implement in Javascript, all you have to do is take an existing json object and store it like this: localStorage.setItem('dysonSwarmGameState', JSON.stringify(gameState));

Each time the game loop runs, it calculates resource gains and losses based on the current gameState. All the other action happens when the buttons are clicked (these change the gameState). Since there are currently 56 of these buttons that get unlocked at various stages of the game, the buttons have their own file called buttonFunctions.js. The file contains metadata about what the buttons do along with functions that get invoked when the buttons are clicked. The button functions get attached to the actual button DOM elements when the game is instantiated, by looping through all the button elements and running handleButton() on them when the game is loaded. As buttons are clicked, new stages of the game are unlocked, until you've built a solar-system spanning civilization.

The most complicated part of building the game was creating and importing the animations, particularly the satellites.js animation. I initially made them with SVG, but for the orbital animation things started to break down at around two thousand satellite elements. I ported this to Canvas, since it can more efficiently render a greater number of elements, being raster-based instead of vector-based.

Another tricky thing was covering all the corner cases for the game logic. Some elements (like "sell shipyard") were only implemented after friends played through the game and got stuck at various points. There's still some balancing I could do to make the game flow better, and I have a list of small improvements in the repo that I'd like to make. Managing game state properly is difficult!

Using Q Developer

One of the elements of the AWS Game Builder Challenge was to use AWS Q Developer to help build the game. I was excited to try it out, since I've used other coding assistants to great effect in the past. I downloaded the VSCode Q Developer plugin and used it that way.

The pros:

The built-in chat interface is good, you can ask it questions as you go along and it will provide useful answers.
The /dev feature will actually create a diff and also apply it to your code if you approve it! This prevents tedious copy/paste operations and saves time.
Q Developer is very good at replicating repeating patterns in code. For instance, I told it things like "add 2 percent inflation to all repeatable buttons (other than the marketer button), in the same pattern as the ad purchase button inflation." and it saved a ton of time!

The cons:

The /dev feature takes a long time (on the order of minutes), because it does multiple LLM invocations on the back-end.
The /dev feature is good at generating code diffs is good for some things, but it often decides to create new files instead of adding code in-line. You have to be very careful in your prompt management to keep it from going off the rails.
I think Q Developer has room for improvement, but AWS is rapidly iterating on it and I foresee the tool being used more often in the future.

The game is open-source!

I've released the entire source code on GitHub under an MIT license, so please feel free to use any of the elements as a base for your own game.

If you end up building a game of your own, I'd love to hear about it!

Vendor lock-in when using AWS Graviton processors is no longer a real thing

Joe Stech — Sun, 29 Sep 2024 23:15:23 +0000

We're entering a golden age of Arm-based processors. It used to be that when you decided to save 20+% on your cloud compute bill with Graviton processors, purists could come out of the woodwork and tell you "you're locking yourself in to AWS by compiling your software for Graviton! You're ensuring that you won't be able to move your most critical workloads ever." I actually heard from one of these people today, which provoked me to write this rant.

In 2024, it is absolutely no longer the case that compiling for Graviton locks you in to anything. Leaving aside the fact that Arm-based Ampere chips have been available on many major clouds for a while (including Azure and Google Cloud), with the announcements of Azure's Cobalt chip and Google's Axion chip, the number of options available to run Arm-based workloads in the cloud has exploded.

These new chips are all built using flavors of Arm Neoverse, making it almost certain that your Arm-based workloads will port with no issues, unless you're using very specific processor features (in which case, you knew exactly what you were doing when you used those features in the first place).

At this point it's a no-brainer to use Graviton for your AWS compute workloads, unless you're using some arcane x86 features. You're going to get significant savings across the board, which will likely only get better as competitive pressures mount. You'll also reduce energy use significantly, which seems poised to become even more important in this era of energy-guzzling GPU clusters.

At this point using Graviton for your compute workloads is all upside, and anyone telling you different is living in the past.

A serverless Python app to send public domain gutenberg.org ebooks to your Kindle with one click!

Joe Stech — Sun, 19 Feb 2023 00:52:04 +0000

I wrote a little serverless web app that does the following:

Downloads a page from gutenberg.org (Project Gutenberg is a non-profit project to provide free public domain ebooks to the public)
If the url is a book download page, adds a column to the "download this ebook table" which contains a link and an input for an email
After the user puts their kindle email in the input field and clicks the "send to kindle" link, the ebook is sent to the user's kindle!

This was a super fun project that I'm going to walk you through from beginning to end. It uses AWS API Gateway, AWS Lambda, and AWS SES. Before we start digging into the details of how I built it, you can try it out by adding "joe@compellingsciencefiction.com" to your Kindle safe sender list and then adding any gutenberg.org path to https://sendtokindle.compellingsciencefiction.com like this:

~~https://sendtokindle.compellingsciencefiction.com/ebooks/67368~~

My little app (which is hosted at a subdomain of my science fiction site) will add the extra "send to kindle" column into the Gutenberg table, you just have to put your kindle email address in the little box and click the send link.

2025 EDIT: AI scraping bots have been hitting this endpoint so much that I decided to shut it down to save resources. If you'd like to deploy your own, I have the full code below.

Serverless architecture

The architecture of the app is simple, just an API Gateway that invokes a Lambda. The Lambda saves some metadata about what book was downloaded and when in S3, downloads the requested ebook from Gutenberg, and sends it to the specified email address. Here's the CDK code that generates the AWS resources:

from aws_cdk import (
aws_lambda as lambda_,
aws_apigateway as apigw,
aws_iam as iam,
aws_ecr as ecr,
aws_certificatemanager as acm,
aws_route53 as route53,
Duration,
Stack)
from constructs import Construct
import os

class SendToKindleStack(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # execution role
        lambda_role = iam.Role(self, id="sendtokindle-lambda",
            role_name='SendtokindleManagementRole',
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
            managed_policies= [
                        iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AWSLambdaVPCAccessExecutionRole"),
                        iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AWSLambdaBasicExecutionRole"),
                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonS3FullAccess"),
                        iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSESFullAccess"),
                    ]
        )

        repo = ecr.Repository.from_repository_name(self, "SendToKindleRepo", "sendtokindlerepo")

        sendtokindle_management_lambda = lambda_.DockerImageFunction(self,
            "CSFsendtokindleManagementLambda",
            code=lambda_.DockerImageCode.from_ecr(
                repository=repo,
                tag=os.environ["CDK_DOCKER_TAG"]
                ),
            role=lambda_role,
            timeout=Duration.seconds(30)
        )

        api = apigw.LambdaRestApi(self,
            "csf-sendtokindle-management-endpoint",
            handler=sendtokindle_management_lambda,
            default_cors_preflight_options=apigw.CorsOptions(allow_origins=["*"])
        )

        custom_domain = apigw.DomainName(
            self,
            "custom-domain",
            domain_name="sendtokindle.compellingsciencefiction.com",
            certificate=acm.Certificate.from_certificate_arn(self,'cert',<cert arn str>),
            endpoint_type=apigw.EndpointType.EDGE
        )

        apigw.BasePathMapping(
            self,
            "base-path-mapping",
            domain_name=custom_domain,
            rest_api=api
        )

        hosted_zone = route53.HostedZone.from_hosted_zone_attributes(
            self,
            "hosted-zone",
            hosted_zone_id=<zone id str>,
            zone_name="compellingsciencefiction.com"
        )

        route53.CnameRecord(
            self,
            "cname",
            zone=hosted_zone,
            record_name="sendtokindle",
    domain_name=custom_domain.domain_name_alias_domain_name
        )

I've put placeholders in a couple of these fields for privacy, but this is verbatim the CDK code I deployed.

As you can see, the longest part of this IaC is the DNS code! If you don't care about a custom domain and just want to use the API Gateway courtesy domain, you don't even need anything past the Gateway specification.

The Lambda Python code

This Lambda pulls double-duty -- it both returns the gutenberg.org HTML (with a modified table), and it also emails the ebook to Kindle if the request sent is to download an epub image. There are many improvements that can be made, but this quick and dirty Lambda works perfectly for what I need.

I'd like to point out a few fun things about this code:

It downloads the ebook to an in-memory BytesIO object instead of a file, and then builds a MIME attachment with the in-memory file.
The code isn't very robust about how it differentiates between calls -- there are a lot of ways to break this Lambda, in which case it'll just return a generic 502 server error.
The code uses AWS SES to send emails via boto3. Be careful when using SES in this way, it's definitely possible for someone to try spending a ton of money on my AWS account by spamming SES sends. You can set up limits on SES (and API Gateway) to mitigate attacks like this.
I use BeautifulSoup to navigate the gutenberg.org HTML to add my column to the download table. I had to introspect the gutenberg.org HTML to find the correct place in the code to insert my column.

from bs4 import BeautifulSoup
import requests
import uuid
import json
import boto3
import time
from email import encoders
from email.mime.base import MIMEBase
from email.mime.application import MIMEApplication
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from io import BytesIO
from urllib.request import urlopen
import urllib

BUCKET = "S3 bucket name str"

def response(code, body):
    return {
            'statusCode': code,
            'headers': {
                'Access-Control-Allow-Headers': 'Content-Type',
                'Access-Control-Allow-Origin': '*',
                'Access-Control-Allow-Methods': 'OPTIONS,POST,GET',
                'Content-Type': 'application/json',
            },
            'body': body
        }


def send_ebook(url, filename, email):
    epubfile = BytesIO()
    print(url)
    try:
        with urlopen(url, timeout=10) as connection:
            epubfile.write(connection.read())
    except urllib.error.HTTPError as e:
        print(e.read().decode())
    from_email = "joe@compellingsciencefiction.com"
    to_email = email
    msg = MIMEMultipart()
    msg["Subject"] = "gutenberg ebook!"
    msg["From"] = from_email
    msg["To"] = to_email

    # Set message body
    body = MIMEText("your book!", "plain")
    msg.attach(body)

    epubfile.seek(0)
    part = MIMEApplication(epubfile.read())
    part.add_header("Content-Disposition",
                    "attachment",
                    filename=filename)
    msg.attach(part)

    # Convert message to string and send
    ses_client = boto3.client("ses", region_name="us-west-2")
    response = ses_client.send_raw_email(
        Source=from_email,
        Destinations=[to_email],
        RawMessage={"Data": msg.as_string()}
    )
    print(response)


def handler(event, context):
    print(event)
    try:
        print(event['path'])
    except:
        pass

    if "epub3.image" in event['path']:
        # this is a request to send an ebook
        path = event['path']
        email = event['queryStringParameters']['email']
        filename = path.replace("/ebooks/","").replace("3.images","")
        send_ebook(f"https://www.gutenberg.org/ebooks{path}", filename, email)
        client = boto3.client('s3')
        upload_id = uuid.uuid4().hex
        payload = {
            "timestamp": time.time(),
            "book_url": path,
            "user_email": email
        }
        client.put_object(Bucket=BUCKET,Key=f'{upload_id}.json', Body=json.dumps(payload).encode('utf-8'))
        return response(200, '{"status":"Sent '+path+' to '+email+'!"}')
    else:
        # return the gutenberg html with added column
        r = requests.get(f"https://www.gutenberg.org/{event['path']}")
        print(r.status_code)
        soup = BeautifulSoup(r.text.replace('"/','"https://www.gutenberg.org/').replace("https://www.gutenberg.org/ebooks","https://sendtokindle.compellingsciencefiction.com/ebooks"),features="html.parser")

        trs = soup.find_all("tr")
        for tr in trs:
            about = tr.get('about')
            if about and 'epub3' in about:
                print(tr)
                epubpath = f'{about.split("ebooks")[1]}'
                soup.append(BeautifulSoup(f"""
                <script>
                function buildlink() {{
                  let text = document.getElementById("kindleemail").value;
                  document.getElementById("csflink").href = "{epubpath}?email=" + text;
                }}
                </script>
                """))
                tr.append(BeautifulSoup(f"<td><a id='csflink' href='/{epubpath}'>Send<br>to<br>kindle<br>email:</a><br><input type='text' id='kindleemail' oninput='buildlink()'></td>", "html.parser"))
            else:
                tr.append(BeautifulSoup("<td class='noprint'></td>", "html.parser"))
        return {
            'statusCode': 200,
            'headers': {
                'Access-Control-Allow-Headers': 'Content-Type',
                'Access-Control-Allow-Origin': '*',
                'Access-Control-Allow-Methods': 'OPTIONS,POST,GET',
                'Content-Type': 'text/html;charset=utf-8',
            },
            'body': soup.prettify()
        }

If any of this inspires you to create your own little utility web app, please tell me about it (you have my compellingsciencefiction.com email address now :). I love hearing about projects like this.

Commit to your team's development conventions or get out

Joe Stech — Tue, 14 Feb 2023 17:05:13 +0000

I'm not saying this for the good of your team, even though it will be good for your team. I'm telling you this for your own sake: don't martyr yourself for your software development opinions!

Software engineering attracts a certain type of mind. Our thinking is necessarily precise -- the computer does exactly what we tell it to do, and if we tell it to do something incorrectly, that's on us. In general, this leads to strong views on how development should be done. Every programmer I've met that has two or more years of development experience has already developed strong beliefs about sustainable, maintainable development conventions, and does not hesitate to inform you about them.

This is fine in isolation -- you have your principles, you stick to them, and your code is cleaner and more maintainable as a result. You always use spaces, you use the gitflow workflow when branching in source control, you always promote through dev/stg/prd environments, you use the pytest framework for unit tests, etc etc. You are a paragon of forward-thinking development discipline.

Problems arise, though, when you must work on a project with an engineer who doesn't share your ideology. Yes, I use that word on purpose: "best practices" are ideology, not universal laws. Good software can be written in myriad ways, and so two skilled engineers can have equally valid, yet diametrically opposed, opinions about development practices.

And so we come to the hard truth of my rant: it is better to conform to the development practices accepted by the majority of your team than to try and force them to accept your convictions. Nothing tanks productivity on a team faster than arguing about development practices and tooling. It might be hard for you to accept, but your pet convention is not so superior that it is worth wasting engineer-weeks (or even engineer-days!) arguing about.

If you just can't accept the idiotic processes your team has in place, and you feel so strongly about your development conventions that you find yourself trying to convince your coworkers again and again to change their ways, it's probably best for everyone if you try and find a role on a new team that shares your convictions. You will be happier, your former team will be happier, your new team will welcome you with open arms, you will crush all your performance reviews, and society will be improved by all the code you will be shipping.

Go fast and reduce risk: using CDK to deploy your serverless applications on AWS

Joe Stech — Wed, 28 Dec 2022 19:19:22 +0000

There are a truly absurd number of ways to deploy applications on AWS. Corey Quinn wrote two separate articles wherein he described 17 ways to deploy containers on AWS -- that's 34 different ways just to deploy containers (granted, some of the listed ways to deploy are farcical).

All of these options inevitably spawn fights about the "right" way to deploy on AWS. I'm now going to explain my preferred way to deploy a serverless API, and then you can fight me.

The best way to deploy your serverless API on AWS:

Write your API business logic code in a way that can be run on AWS Lambda.
Write a CDK template in the same language as your Lambda function code and store along with your Lambda function code in source control (I'm going to assume you're using git here, but use whatever source control you want).
Deploy your serverless API from your git commit as a self-contained unit.

There's a lot to unpack here, but the main thing I want convey here is that it is awesome to write your executable code and your IaC templates in the same language.

Your CDK template

So what is CDK? It's the AWS 'Cloud Development Kit', and it basically lets you transpile your language of choice (as long as that language is either JavaScript, TypeScript, Python, Java, C#, or Go) into CloudFormation. This is awesome, because you don't have to deal with a weird domain specific language for your IaC or manage state files for your IaC (looking at you, Terraform).

In the case of a serverless API, these are the main things you'll want in your CDK code:

IAM policies that limit your lambda permissions using the principle of least privilege.
A Lambda resource that you'll deploy your code to.
An API Gateway resource (RestApi) that you'll point at your Lambda.

There are a bunch of other services you could include here (RDS database, dynamoDB, etc etc) if you need them, but that's the beauty of this system -- you're not limited by a specific serverless framework, you can just add arbitrary services to your CDK code as you need them.

Here's a bare-bones version of a serverless CDK stack file in Python (you can also write these files in JavaScript, TypeScript, Java, C#, or Go), so you can see what I mean (everything has the generic name "somestack", which you can replace with a more meaningful name):

from aws_cdk import (
aws_lambda as lambda_,
aws_apigateway as apigw,
aws_iam as iam,
aws_ecr as ecr,
Stack)
from constructs import Construct
import os

class SomeStackName(Stack):
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        # Create the Lambda execution role
        lambda_role = iam.Role(self, id="somestack-lambda",
            role_name='SomestackManagementRole',
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
            managed_policies= [
                        iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AWSLambdaVPCAccessExecutionRole"),
                        iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AWSLambdaBasicExecutionRole")
                    ]
        )
        # Get an existing ECR repo that has a docker image you want to deploy to Lambda
        repo = ecr.Repository.from_repository_name(self, "SomestackRepo", "somestack_ecr_repo")
        # Create the Lambda function
        somestack_lambda = lambda_.DockerImageFunction(self,
            "SomestackLambda",
            code=lambda_.DockerImageCode.from_ecr(
                repository=repo,
                tag=some_image_tag_name
                ),
            role=lambda_role
        )

        # Create the API Gateway
        api = apigw.LambdaRestApi(self,
            "somestack-endpoint",
            handler=somestack_lambda,
            default_cors_preflight_options=apigw.CorsOptions(allow_origins=["*"])
        )

That's all the Infrastructure as Code that you need to create a serverless API! You can learn more from the official CDK developer guide.

The actual development workflow

So now you have your CDK IaC written and you're ready to start incorporating it into a real devops pipeline. What does the day-to-day development process look like?

Make a branch.
Make some Lambda code changes, some CDK changes, whatever, and push your branch.
A git server webhook should trigger a CDK build to test your Infrastructure as Code (IaC). This webhook also runs some tests on your temporarily built environment to make sure you have no regressions.
Merge your branch into main (master, whatever you call it).
Another git server webhook that only triggers on main branch commits deploys your code to your dev environment (which can then be promoted to stage/prod as necessary)

Now you're ready to build serverless, production-grade APIs with incredible speed and very little risk! You don't have to have a team of engineers doing server maintenance, and you can spin up as many independent microservices as you need to without worrying about autoscaling policies or cluster management. It's an incredibly powerful pattern for certain types of software.

'Twas the Night Before Christmas: Cloud Edition

Joe Stech — Tue, 20 Dec 2022 15:33:37 +0000

'Twas the night before Christmas, and all through the cloud
Every server was stirring, it was really quite loud.
The lambdas were running, provisioned with care,
In hopes that complex workloads soon would be there.

The devs were done, nestled all snug in their beds,
While visions of SageMaker danced in their heads.
And mamma in her t-shirt, and I in my briefs,
Had just settled down for a long winter's sleep.

When out in the cloud there arose such a clatter,
I sprang from my bed to see what was the matter.
Away to the keyboard I flew like a flash,
Tore open the console and threw up the dash.

The screen on the desk (what a stupendous glow)
Gave the luster of mid-day to keyboard below.
When, what to my wondering eyes should appear,
But a miniature vid, streaming beautifully clear.

With a middle-aged speaker, but such a fast learner,
I knew in a moment it must be Dr. Werner.
More rapid than Redis his new features came,
And he whistled, and shouted, and called them by name:

"Now, Lambda! now, CloudWatch! now, S3 and Redshift!
On, CloudFront! on, Kendra! on, Neptune and GameLift!
To the top of the bill! Let it never be small!
Now more dashboards! more dashboards! dashboards for all!"

He clicked the last slide, to his team gave acclaim,
I'm not kidding! He even called out their full names.
Then I heard him call out, ere he dove out of sight,
“Happy holidays all, and to all a good-night.”

Adding an existing custom Lambda Authorizer to API Gateway with CDK and Python

Joe Stech — Thu, 15 Dec 2022 16:33:21 +0000

OK, this is an pretty niche one, but I had to write it up because it annoyed me so much figuring it out. I'm using the Python CDK library, aws_cdk, not the TypeScript interface (but the concepts are the same).

If you have an existing custom Lambda Authorizer, and you want to add it to a new API Gateway LambdaRestApi, you can't just grab the Authorizer with a normal from_XXXX command in the Authorizer CDK class. Such a from_ function doesn't exist for Authorizers, as of CDK 2.20.0.

What you have to do is specify the Lambda function used by the authorizer, using the from_function_name function. You can then create a TokenAuthorizer based on your existing authorizer Lambda, and pass that to your LambdaRestApi, like so:

from aws_cdk import (
aws_lambda,
aws_apigateway)

# ...
# create your stack class, IAM roles, etc

# define some EXECUTION (not auth) Lambda for your API, here we'll call it api_lambda

# grab your existing Lambda auth function from the lambda name
auth_function = aws_lambda.Function.from_function_name(self, "myAuthLambda", "existing_lambda_name")

# New Authorizer based on your existing Lambda
lambda_authorizer = aws_apigateway.TokenAuthorizer(self, "myAuthorizer", handler=auth_function)

# Your new Lambda-backed and authorized API
api = aws_apigateway(self,
        "myEndpoint",
        handler=api_lambda,
        default_method_options={"authorizer": lambda_authorizer, "authorization_type": aws_apigateway.AuthorizationType.CUSTOM})

And that's it! Now you've got an API Gateway that uses a Lambda Authorizer and also a Lambda execution backend.

If you're still running into issues with your specific setup, it's pretty easy to do development testing of your Lambda Authorizers. In the API Gateway console, first go to the endpoint that calls your authorizer, and then click "Authorizers" in the left nav bar. Each Authorizer will have two links at the bottom of their panes: "Edit" and "Test". Clicking "Test" is going to be your best bet at getting actionable logs.

Solving problems with API Gateway and the Python CDK

Joe Stech — Fri, 09 Dec 2022 16:53:00 +0000

So you decided to use the Python flavor of CDK to build your API Gateway infrastructure. Great! Python CDK is my favorite tool to manage Infrastructure as Code in AWS. Anyone who knows Python and has created resources in the AWS console can get up to speed almost instantly with Python CDK. But now you've hit an incredibly hard to debug issue: you've specified an existing API Gateway in a different CDK Stack, and are now trying to use aws_apigateway.LambdaRestApi.from_rest_api_attributes to get the existing API Gateway and add new paths to it using add_resource and add_method.

Everything is going great, your new stack deploys, and then you try to invoke your new API Gateway->Lambda endpoint. You get

Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a Date' header.

Oh no. That generic error could mean anything. So you Google around for a while, trying to figure out what could be going wrong. You look at the API Gateway in the AWS console. You realize that even though the resource paths are being added to your Gateway, they're not being deployed to a Stage!

The LambdaRestApi class should do that automatically, right? Yes, it will, but unfortunately not if you're using the from_rest_api_attributes function to get an existing Gateway. This is a known issue: https://github.com/aws/aws-cdk/issues/12417

I'd encourage you to upvote the github issue above so that the CDK team prioritizes it, but here's the workaround with Python CDK:

import uuid
from aws_cdk import aws_apigateway

api = aws_apigateway.LambdaRestApi.from_rest_api_attributes(self, [gateway id], rest_api_id=[api id], root_resource_id=[gateway root resource id])
r = api.root.add_resource("hello-world")
r.add_method("ANY", aws_apigateway.LambdaIntegration([your previously specified Lambda CDK object])
deployment = aws_apigateway.Deployment(self, f"deployment-{uuid.uuid4().hex}', api=api)
stage = aws_apigateway.Stage(self, [stage id (some string)], deployment=deployment, stage_name = [some string])
api.deployment_stage = stage

And now your new endpoint deploys properly. As you can see, there are many parameters that you need to create yourself (I've put those in square brackets), but the main thing to note is that I've manually added a Deployment and Stage to the API Gateway, and added a hash to the deployment ID that changes every time the stack is deployed. This forces the deployment to be pushed out, and the solution won't work without it.

If you need to add multiple resources/methods in different stacks (my own current use case!), the best way I've found is just to put the deployment into a separate stack and build it after you add/update any of your resource stacks. It's a little gross, but prevents you from having to add multiple stage names.

If this one had you beating your head against a wall for a while, you're not alone. I hope the CDK team addresses this issue soon, but in the meantime you now have a fix.

How I built my ideal daily Python newsletter with AWS and Python

Joe Stech — Thu, 08 Dec 2022 17:00:40 +0000

I'm a heavy Python user. I've basically used it to do my job every day for the last decade, so it's valuable for me to keep tabs on new developments in the Python ecosystem. Amazing programmers talk about new Python developments on Hacker News, Reddit, and Twitter all the time, so I created a daily newsletter to collect the top Python stories from all three platforms. If you want to take a look at the finished product, it can be found here: compellingpython.com

Secret weapon number one: AWS Lambda with Docker

Using Docker containers on Lambda has been a little bit of a game-changer for me, mainly because it's so easy to add extra supporting stuff (Jinja HTML template files, packages, etc) to the container without having to have an elaborate custom build system.

Sending a daily email is a perfect application for AWS Lambda, because all it takes is a single invocation that runs for a max of 15 minutes a day (using 128 mb of RAM). This comes out to about 6 cents per month (yes, you read that correctly, 6 cents per month), which is essentially free.

With Docker Lambdas, all I have to do is write a Dockerfile that looks like this:

FROM public.ecr.aws/lambda/python:3.9

COPY requirements.txt .
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

COPY hn.py reddit.py twitter.py article_processor.py app_handler.py ${LAMBDA_TASK_ROOT}/
COPY templates/ ${LAMBDA_TASK_ROOT}/templates/

CMD [ "app_handler.handler" ]

This will install the python packages I specify in a requirements.txt file, then copy over my python source files for reading data from HN, Reddit, and Twitter, along with the main Lambda app_handler.py file, and then tell the Lambda container where the entry point function is (a function called handler in the app_handler.py file.

After I build and push the docker container to ECR (instructions for this can be found right in your ECR repo!), then it's time for me to deploy the Docker image to Lambda with AWS CDK.

Secret weapon number two: AWS CDK

Infrastructure as Code – so valuable, yet so tedious. In the past I would always groan inwardly when it was time to write Terraform or Cloudformation to template out my AWS resources, but in 2019 Amazon released CDK. CDK supports Python, so I can now write all my IaC in Python, which is a joy! Python CDK code transpiles to Cloudformation (with an intermediate Typescript step, more on that later), so all the state information lives in AWS and I can use drift detection on my stacks. I know this is a relatively new tool, but I'm surprised it's not used more widely.

So the first step in building the newsletter is deploying the Docker Lambda I discussed above using CDK. This Lambda will obtain data from Hacker News, Reddit, and Twitter, filter it, and then send me a nicely formatted email.

To provision the Lambda along with an IAM role, this is all the CDK code I need:

lambda_role = iam.Role(self, id="python-newsletter-lambda",
    role_name='PythonNewsletterRole',
    assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
    managed_policies= [
                iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AWSLambdaVPCAccessExecutionRole"),
                iam.ManagedPolicy.from_aws_managed_policy_name("service-role/AWSLambdaBasicExecutionRole"),
                iam.ManagedPolicy.from_aws_managed_policy_name("AmazonS3FullAccess"),
                iam.ManagedPolicy.from_aws_managed_policy_name("AmazonSESFullAccess"),
            ]
)

repo = ecr.Repository.from_repository_name(self, "NewsletterRepo", "python_newsletter")

newsletter_lambda = lambda_.DockerImageFunction(self,
    "PythonNewsletterLambda",
    code=lambda_.DockerImageCode.from_ecr(
        repository=repo,
        tag=os.environ["CDK_DOCKER_TAG"]
        ),
    role=lambda_role,
    timeout=Duration.minutes(15)
)

There are three pieces here:

Generating the IAM role. I'm not being super locked-down in this example, since I'm giving the Lambda full S3 access and full SES (Simple Email Service, more on that in the next section) access. You can generate custom locked-down policies here instead of using the AWS-managed policies.
Creating a reference to an ECR repo. ECR is the Docker image repo service that Amazon provides. This is the repo that I pushed my Docker image to in the first section of this article. If you're new to CDK, I want to point out that if you're referencing existing resources you should always use the from_* function from the CDK documentation about the resource you want to use. Check out what I'm talking about in the CDK ECR documentation here.
Deploy the Docker Lambda. You can see that all I had to do was specify which image tag in the ECR repo I want to deploy (via the CDK_DOCKER_TAG environment variable), specify the role I created, and specify that the timeout is 15 minutes.

That's it! Now when I run cdk deploy my Lambda will be deployed with all the proper configuration, and if I ever need to change anything I just have to rerun cdk deploy. The final step is just to schedule the Lambda daily using a cron job in EventBridge.

Secret weapon number three: Python and AWS SES

OK, I acknowledge that Python is about the least secret weapon there is. But now that I've described the infrastructure, you might be wondering what's inside this magical little email-sending Lambda function. I'll leave the tedious API calls to get the data from HN/Reddit/Twitter as an exercise for the reader, but I do want to talk briefly about the design of the code package – particularly how the emails are sent.

Here's the basic process:

First, I grab all the data from the community APIs for the previous day and rank by upvotes.
Next I grab the top 3 posts about Python from each community, and grab the full HTML of the web page.
After parsing out the main content of the page (doesn't always work), I send the main body text to my summarizer algorithm (post about that later!) and get a summary of the article.
I take all the metadata about the top three articles for each community and use it to populate a Jinja2 HTML template.
Finally, I send out the HTML emails to myself (and my discerning friends and colleagues who have signed up at compellingpython.com) using AWS SES!

SES (Amazon's Simple Email Service) is fantastic for an application like this, because it's just about the cheapest way to send bulk emails programmatically that there is. You can send 62,000 emails a month for free, and after that it's 10 cents per thousand. Ten dollars to send a hundred thousand emails without maintaining all your own SMTP infrastructure is kind of a ridiculous deal.

There are some drawbacks, like having to carefully manage your own deliverability metrics and email lists (most other hosted email services provide these things) but as a simple email sending utility I have no complaints. If you want to pay more you can get dedicated IPs, but for fun projects like this one I just use the shared IPs that SES provides for free.

That's it! That's the service. There is one other aspect to this whole newsletter that I haven't discussed, which is the collection and management of email addresses themselves – I have a separate Lambda function for this that I might write up at a later date. For now though, take care and have a great week.

Notes from AWS re:Invent 2022: the Giant Magellan Telescope

Joe Stech — Mon, 05 Dec 2022 16:15:13 +0000

Most of the sessions I attended at re:Invent this year were highly technical, but I also attended some broader science talks, and this talk by Giant Magellan President Dr. Robert Shelton was one of them.

Apparently in the Atacama Desert, the driest place on earth, a new telescope is being built: the Giant Magellan Telescope. It's going to be 22 stories tall, the entire building will rotate, and will be up to 200 times more powerful than existing research telescopes. https://www.giantmagellan.org

The telescope's president was presenting at AWS because AWS provides the compute required to simulate the telescope's building envelope. The telescope will require precise environmental controls, and the Atacama Desert is a place with wild temperature swings. With AWS-based simulation, the Giant Magellan project can ensure that their building designs will support the environmental controls required by telescope and the extraordinarily sophisticated instrumentation that can be attached to the telescope!

Dr. Shelton spoke about how the telescope will have the capability to resolve atmospheric composition of exoplanets at the level required to detect extraterrestrial life, finally answering the age-old question: are we alone in the universe?

Notes from re:Invent 2022: a serverless/devops recap

Joe Stech — Sat, 03 Dec 2022 00:28:01 +0000

This week was my first time attending re:Invent, and it was as intense as I had imagined! With 60k people flying into Vegas for the conference and dozens of events happening every single hour of the day, picking and choosing what to spend time on was a challenge. I got a great overview of the new (and some existing) serverless and devops tools, though, and met some awesome new people, which was my primary goal. I'll write more detailed articles about select sessions next week, but here were my favorites, with links to YouTube so you can watch them too!

Building containers for AWS (CON325)

https://www.youtube.com/watch?v=S7JwFFZ-7_Q

If you build Docker containers locally on macOS, this was a talk for you. Phil Estes and Jesse Butler did an entertaining job of presenting a new open-source alternative to Docker Desktop on macOS called Finch.

Check out the talk for details, but my main takeaways were:

Finch is a macOS native client for container development.
Finch uses the same interface you're all familiar with for building containers (i.e. finch build -t [name:tag] . instead of docker build -t [name:tag] .)
Use the --secret parameter to pass in your secrets to the container on build, and then use --mount-type=secret when you run. Don't put secrets in layers and delete them later! They're still there, embedded in the previous layer.
Use multistage builds to make your containers MUCH smaller.

Deep dive into Amazon Neptune Serverless (DAT322)

https://www.youtube.com/watch?v=WwuAfTpL7ww

Not a lot of developers use graph databases, but since basically everything in the real world can be represented as a graph, they're incredibly useful! And now you can run a Neptune graph database without allocating servers (kind of). Takeaways:

You can't scale down to zero, the minimum your cluster can go is to 2.5 NCUs. This is about $290/month at $0.1608/hr per NCU, which is pretty pricey for an absolute minimum serverless setup. They're working on lowering minimums.
Can only scale up to about the size of a r6g.8xlarge
Is not yet supported by CloudFormation (I am constantly surprised by how poorly AWS supports CloudFormation).

Practical experience with a serverless-first strategy at Capital One (SVS311)

This one hasn't been uploaded to YouTube yet, but it was one of my favorites of the conference. They set it up as a back and forth between a very senior engineer at Capital One and an AWS architect, both explaining their own recommended best practices for Lambdas at scale (Capital One uses a LOT of Lambdas). Takeaways:

Use Lambda init (cold start) to cache secrets for subsequent calls to the same warm Lambda
Lambda init gives you a whole CPU (no fractional throttling) no matter what your allocated Lambda size is. This means you can do "Lambda power tuning" to optimize for cost/performance
If you get errors in your init phase, this will result in stuck Provisioned Concurrency. AWS won't retry, you'll need to release a new version

How AWS Application Composer helps you and your team build great apps (SVS211)

Again, no upload yet for this one. AWS Application composer is basically a visual yaml editor for CloudFormation templates.

I'm looking forward to using this to visualize existing architectures already built in CloudFormation, but right now the tool is pretty limited (it's still in Preview). Takeaways:

The service is very slick for quickly and visually compiling new resources into a yaml
It doesn't hook directly into CloudFormation state, you have to upload the actual yaml file if you want to see an existing architecture
There are a bunch of services that it doesn't support out of the box (it doesn't error when loading such a yaml, it just grays out the unsupported services).
Basically it's just a yaml visualizer/editor for now. The service is promising though!

Governance Guardrails across your multi-account environment (COM312)

A great dev talk by Paushali Kundu about how Thomson Reuters manages compliance enforcement and monitoring across their 300+ AWS accounts.

I learned about CloudFormation Hooks for the first time, which can enforce rules on CloudFormation stacks.

Accelerating workloads using parallelism with AWS Step Functions and Lambda (API205)

Yet another one with no uploaded video. I was initially skeptical about map/reduce with Step Functions and Lambda, only because there are already so many great tools to do massively parallel data processing. I was pleasantly surprised though!

If you have millions of CSVs (or whatever) on S3 that you want to process in parallel, they've made it SO EASY to do so. Basically all you have to do is use a visual editor on the AWS console to make a mapping of S3 objects->Lambda function mapper, and then plug in a final reducer lambda, and you're done. They did a live demo where they processed 550k csv files on S3 in about 2 minutes, it was awesome. Takeaways:

If you don't want to set up your own cluster to do your map/reduce calculations, absolutely use Step Functions
Concurrency limits can be tuned so you don't overwhelm externally-called services that have API limits
You can easily specify how many files to batch in a single Lambda by the number of items OR a number of kilobytes
You can set a number of items that can fail before the whole thing fails, OR you can set a percentage of map invocations that can fail
You can send the compiled results from the mappers directly to S3

That's it for now, I have to catch my flight! I'll be writing more about re:Invent next week.