Forem: Peter McAree

Letting Kiro Drive — Autopilot and Hooks

Peter McAree — Mon, 12 Jan 2026 20:57:48 +0000

Following on from my previous post, I thought I would dive a little deeper into two specific features with AWS Kiro that I use frequently. Two features that I feel work well for my use cases.

As I build trust with agentic workflows, I find myself delegating more tasks (of specific types) to Kiro's agent. That trust is building through the spec-driven development nature of Kiro, and the approach that it takes.

Tasks are broken down into small, atomic pieces of work that have been defined and reviewed up front. This approach makes it much more comfortable to enable "autopilot" mode with Kiro, in combination with hooks to allow the agent to handle these tasks with confidence.

What Autopilot Actually Does

AI-driven software development workflows have came a long way from just a glorified autocomplete. The notion of allowing the agents to take the wheel, and have full control over tasks has became the norm and this has resulted in some higher-level observability and review settings:

Autopilot mode is the more autonomous mode where Kiro can take a higher-level goal (like "add tests" or "refactor this module") and then propose the concrete edits to get there.
Supervised mode keeps the same underlying engine but requires stepwise approval, so you review changes in smaller slices instead of one giant diff dump.

Under the hood, autopilot leans on Kiro's spec‑driven workflow: requirements, design, tasks, and the eventual diffs are all linked, which makes it possible to trace a change back to its origin.

Agent Hooks in Day‑to‑Day Dev

In contrast to autopilot, agent hooks are the little agents that quietly keep your house in order.

Hooks are event‑driven triggers e.g. on save, on file created etc.
Kiro also supports manual hooks using the userTriggered type, which you can fire on demand when you want a specific workflow to run.
Common patterns include updating tests or documentation whenever a file changes, or running lint/security checks automatically on certain paths.

I think of hooks as new-gen scripts (Bash, PowerShell etc.) - very similar to pre/post build, compile, or lint scripts but more intelligent given the context that they can consume and act upon.

A Tiny Hook With Massive Impact: Conventional Commits

Not every use of Kiro (or agentic workflows as a whole for that matter) needs to be a big refactor. Some of the highest-leverage changes are the ones that shave 30 seconds off tasks you do dozens of times a day.

One of my most frequently used hooks in this setup is a manual, user-triggered hook that generates Conventional Commit style messages for the currently staged changes:

This uses Kiro's userTriggered hook type, which makes the hook available as an on‑demand action in the IDE, which sends a structured prompt plus the current Git diff to the agent.

The prompt itself encodes the Conventional Commits rules - type(scope): description plus an optional BREAKING CHANGE footer - so in practice the workflow becomes:

Stage changes as usual.
Trigger the "Conventional Commit Message" hook.
Let Kiro propose something like feat(api): support filtering by status or fix(auth): handle expired refresh tokens gracefully, complete with a breaking-change footer when necessary.

The result is that the repo gets consistently structured commit history, semantic versioning stays sane, and nobody has to mentally juggle the Conventional Commits spec while in the middle of a refactor.

A real example of this that I have used for one of my projects can be seen below:

This is a nice microcosm of how hooks work best: encode a convention once in a hook, and then reuse that decision every time with a single trigger.

Setting Boundaries For Autopilot

Letting an agent touch many files at once is liberating until it starts sprawling changes in areas that you might not have anticipated updating as part of the current item of work. This is where autopilot mode can begin to get out of hand, so it's important to ensure that boundaries are clearly defined and set.

A few patterns that kept things sane:

Scope autopilot by intent: As with all AI-driven development work, a significant amount of importance is on the prompt. With Kiro's spec driven nature, I find it really easy to explicitly scope autopilot per task which works great for me.
Prefer supervised flows for risky areas: Infrastructure, auth, and anything that touches external contracts are better run in supervised mode to force smaller, reviewable steps. For areas like this, I typically find myself using the agent as more of a Q&A partner rather than allowing it to control the direction.
Align hooks with existing policies: If your team already has rules like "integration tests for new endpoints" or "docs for public APIs", encode those as hooks instead of inventing new, opaque automation that nobody can reason about.

The goal isn't to stop Kiro from making mistakes whenever you give it the keys, but to ensure those mistakes are easy to spot, easy to revert, and clearly tied to an explicit intent rather than vague prompts.

Kiro Take the Wheel: Pros and Cons

There's a genuine trade-off in letting an autonomous agent loose on your production codebase. This isn't a demo branch; the changes need to age well.

Where Kiro shines

Velocity: Multi-file transformations, test generation, and doc updates happen in one pass at the time of writing rather than coming back to it in the future - even if it's a PoC!
Fewer boring tasks: The likes of boilerplate, and glue code, move out of your head and into hooks and autopilot tasks.
Consistency: Hooks enforcing tests, docs, checks, and even commit message formats across the project helps to drive consistency.

The costs you have to own

Review fatigue: Large, agent-generated diffs are tiring to read - even if they are scoped to an individual Kiro spec task. Without discipline, they encourage rubber-stamping.
Over‑eager changes: autopilot will happily assist in adjacent areas if your intent is vague, which can introduce churn or subtle behaviour changes that may result in unintentional bugs.
Trust gaps: You absolutely cannot blindly trust the code that the agent generates - it needs to be reviewed. This goes hand-in-hand with the first point around fatigue here.

Healthy use of Kiro looks less like fire and forget, and more like pairing with another engineer: give clear instructions, review carefully, and never merge something you wouldn't defend in a post-incident review.

Observability Playbook

If you're going to let Kiro drive on real work, a lightweight playbook helps keep both your repo and your attention span intact.

Start in supervised mode on critical areas. Switch to autopilot for well-scoped, low-risk improvements like adding tests or aligning docs.
Use hooks to automate what you already require (tests, docs, checks, Conventional Commits).
Tie every autopilot run to a clear spec and make that spec part of the PR description, so reviewers see the why as well as the what.
Keep CI, static analysis, and policy gates in front of production; agents make it easier to pass them, not optional to have them.

Done well, you end up with a workflow where Kiro handles the grind - tests, docs, type tightening, commit hygiene, repetitive refactors - while you stay responsible for intent, boundaries, and everything with real blast radius.

There is no doubt that it requires a shift in both mindset and hands-on working approach, but trying it out and being open to said changes might unlock some efficiencies in how you work on a daily basis.

Resources

If you're interested, the full JSON definition for my conventional commit hook is as follows:

{
  "enabled": true,
  "name": "Conventional Commit Message",
  "description": "Automatically generates a commit message following conventional commit standards based on the current git diff",
  "version": "1",
  "when": {
    "type": "userTriggered"
  },
  "then": {
    "type": "askAgent",
    "prompt": "Review the current git diff and write a commit message following conventional commit standards. The format should be: type(scope): description. Common types include: feat, fix, docs, style, refactor, test, chore. Keep the description concise and in present tense. If there are breaking changes, include BREAKING CHANGE in the footer."
  }
}

Give it a try, and let me know what your experience with it is!

Kiro's Agentic IDE: Hype, Hope and Hard Truths

Peter McAree — Thu, 18 Dec 2025 23:39:15 +0000

With AI tooling absolutely dominating the tech space (and world for that matter) right now, it only seems fitting that as an engineer I test the bleeding edge tools to evaluate for myself. So I thought I'd give it a test whilst building a real feature that was sitting in my backlog for a while, but never had the time to get round to.

The feature in question is a embedded social widget within an existing blog website. And delving into Kiro and walking through its initial setup of how it begins to craft a feature started to make me shift my mindset about agentic coding capabilities: the magic isn't in how fast you can generate code, it's in how well you can think through what you're building before a single line gets written.

Let me walk you through what I think makes this IDE different, and why it might just change your development workflow too.

What is it?

Before diving into my experience, let's establish what Kiro actually is. Developed by AWS and launched sometime in summer 2025, Kiro is an agentic IDE that introduces a structured methodology called "spec-driven development" to AI-assisted coding. Unlike traditional AI coding assistants that jump straight from prompt to code, Kiro forces you to think through three distinct phases:

Requirements: What you want to build and why
Design: How it should be architected and implemented
Tasks: Discrete, sequenced implementation steps

Similar to other offerings currently on the market, Kiro is a fork of VSCode - so for someone who primarily develops using TypeScript, it feels familiar.

There are some additional context artifacts that Kiro generates, and bakes into the prompts (e.g. steering docs), but I'm not going to cover those aspects in this blog - possibly at a later date!

The Spec-Driven Development Workflow

The key differentiator in my opinion, is the spec-driven development aspect of the product. Instead of purely allowing the AI to churn lines of code out, sometimes dubbed "vibe coding", Kiro implements a rigorous, three-phase approach that I found great to interact with.

Phase 1: Requirements Generation

When I started building the embedded social widget, I simply typed:

Create Spec: Embedded tweets/YouTube videos within the blog posts

Kiro immediately transformed this single prompt into a comprehensive requirements document using EARS (Easy Approach to Requirements Syntax) notation.

Within minutes, I had:

User stories for viewing cards, and managing interactions
Acceptance criteria for edge cases I hadn't considered
Explicit assumptions that made it clear what the system would do (and would not do!)

With some of the context that Kiro inferred from the project, it was able to create some pretty comprehensive user stories that aligned with my overall goal of building the feature of embedded social media post support.

This phase alone was invaluable as I deliberately kept the initial prompt lightweight to truly understand how deep (or vague) Kiro would take it. It forced me to think about what I actually wanted to build, not just start coding and figure it out later. I could iterate on the requirements, refining them until they captured exactly what I needed.

Phase 2: Technical Design

Once I approved the requirements, it continued to analyse my existing Next.js codebase and generate a complete technical design document. This included:

Data flow diagrams showing how information moves through the widget
TypeScript interfaces defining the shape of my data structures
Markdown API with clear contracts

This definitely was the most interesting part for me - this is where it highlighted different edge cases that may not have been immediately on my mind whilst first building the feature from scratch. Things like supporting different flavours of URLs to embed (e.g. short codes). For this particular project, and the scale that it is at, this could be considered overkill, but I quite like that it was able to highlight and include support for this.

Phase 3: Task Sequencing and Implementation

This is where Kiro's approach really shines. The IDE broke down the entire feature into individual, dependency-ordered tasks. Each task included:

What functions, and features it is building
New interfaces that it will design
References to the requirements that it was supporting

Instead of facing an overwhelming project, I had a clear roadmap of small, manageable chunks. I could execute tasks one by one, reviewing the output at each stage. If something wasn't quite right, I could iterate on that specific task without throwing away the entire implementation.

On top of this iterative approach, Kiro also had checkpointing baked in if I ever needed to rollback and restore to a particular point - similar to a snapshot restore.

Love/Hate

Having worked on a number of AI-assisted projects before, where results have widely varied, the task-based workflow feels much better than a glorified autocomplete.

Now of course, it still hallucinates. It still writes bugs. However, the smaller iterative approach that allows you to drive deeper context into top-level requirements feels like you have more control over the output.

Smaller Chunks, Better Control

Each task typically took 5-15 minutes for the agent to complete. I could review the changes, test them, and move on without losing context. Compare this to asking another agentic coding tool to "build an entire authentication system" then trying to debug a thousand lines of generated code.

Easier Iteration

When a task didn't work perfectly, I could simply regenerate just that task with additional context or constraints. The agent already understood the broader requirements and design, so refinements were quick and accurate.

Progressive Testing

Because tasks were small and sequenced by dependencies, I could test incrementally as I built. By the time I finished all tasks, the feature was already mostly working, not a massive integration nightmare waiting to happen.

Real-World Results: The Embedded Widget

So how did it all work out? Well with some additional functionality written, and tested - it now looks like the following:

What Went Well

Upfront clarity: The spec process forced me to think deeper about requirements, and ensured I built exactly what was needed.
Manageable: Smaller tasks and chunks felt much more manageable, and allowed me to provide the deeper context to receive a higher quality output.
Fast iteration: When I needed to adjust the design, updating the spec and regenerating tasks was straightforward.

Challenges and Learning Curve

Kiro isn't perfect. There were a few pain points:

Verbosity: For small tweaks, the full spec workflow can feel like overkill - especially with re-generating tasks. Learning how to strike a balance is key.
Spec maintenance: As your codebase evolves, keeping specs in sync requires discipline.
Availability of models: Kiro currently only supports some of Anthropic's Claude models which is certainly a bit lacklustre in comparison to the like of Cursor.

Key Takeaways from My Experience

After completing my first AI-driven project with Kiro, here's what I learned:

1. Thinking Before Coding Matters More Than Ever

With AI, it's tempting to skip planning and just start generating code. Kiro forces you to slow down and think, and that discipline pays dividends in code quality and reduced rework.

2. Specs Are Living Documents, Not Dead Weight

Unlike traditional waterfall specs that get written once and forgotten, Kiro's specs evolve with your code. They stay in sync and provide ongoing value for onboarding, refactoring, and feature additions.

3. Small Tasks Beat Big Prompts

Breaking work into discrete, reviewable tasks makes AI-assisted development predictable and manageable. You're not gambling on a massive code dump working perfectly.

Summary

AWS Kiro represents a shift in how we think about AI-assisted development. Instead of treating AI as a faster autocomplete, its focus on spec-driven development feels like we have more control over the output, and that works best when given clear direction and structure.

For my embedded widget project, that structure was exactly what I needed. The spec-driven workflow helped me think through requirements, communicate intent clearly, and iterate on discrete chunks of functionality. The result is production-ready code that's well-tested, documented, and maintainable.

My recommendation would be to give it a try. The free-tier that comes with Kiro is enough to build a small feature to get a feel for the approach that it uses. Like I had mentioned previously, it does generate other artifacts that form part of the context that gets supplied to the LLM, but I'll talk about those in a future post.

You can check it out at https://kiro.dev/.

How to build an API Gateway with a custom domain using AWS CDK

Peter McAree — Fri, 05 Sep 2025 09:30:58 +0000

Having a professional, branded API endpoint is essential for modern applications - especially when you want your users to interact with something memorable like api.yourcompany.com rather than the default, randomly generated AWS endpoint like https://d-abc123xyz.execute-api.us-east-1.amazonaws.com.

If you have read any of my previous blogs, you'll know how much I love API Gateway - and this is another reason too! This is particularly nice to keep branding consistent if you are allowing your users to directly consume the API in their own applications.

So let's take a look at how we can provision an API Gateway with a custom domain - all using AWS CDK.

✍️ Define some CDK

Unfortunately, like with many AWS services, CDK doesn't always provide the most elegant L2 constructs for every scenario. However, the API Gateway constructs are quite mature, so we'll be working with a nice combination of L2 constructs and some manual configuration.

Before we jump into the CDK itself, the following examples assume that you have a few things already provisioned and in place:

A registered domain name (can be managed by Route 53 or external registrar)
A hosted zone in Route 53 for your domain

Provision the cert

We must create a new TLS certificate for usage on our API Gateway with our custom domain. The following creates the certificate, and specifies that DNS records are going to be used to verify it:

const domainName = "api.yourcompany.com";

// Certificate for custom domain
const certificate = new certificatemanager.Certificate(
    this,
    "ApiCertificate",
    {
        domainName,
        validation: certificatemanager.CertificateValidation.fromDns(),
    }
);

Provision API Gateway

Now for the main event - creating our API Gateway with a custom domain. There are a couple of approaches here, but I'll show you the most straightforward method using the RestApi construct with domain configuration:

// API Gateway
const api = new apigateway.RestApi(this, "Api", {
    restApiName: "Custom Domain API",
    domainName: {
        domainName,
        certificate,
        basePath: "v1", // Optional
    },
});

// Sample resource and method
api.root.addMethod("GET");

The above provisions a new gateway that consumes the certificate and the custom domain name that we want to specify. Additionally, it also specifies a basePath that simply prefixes the routes that you define on your API gateway - example: /v1/users/.

Finally, it also adds a HTTP GET method onto the root definition of the gateway.

DNS changes

The final piece of the puzzle is configuring the new A record within your Route 53 hosted zone. This will effectively just point to the API gateway and that finalises the alias.

const hostedZoneId = "Z1234567890";

// Route53 alias record
const hostedZone = route53.HostedZone.fromHostedZoneAttributes(
    this,
    "HostedZone",
    {
        hostedZoneId,
        zoneName: "yourcompany.com",
    }
);

new route53.ARecord(this, "ApiAliasRecord", {
    zone: hostedZone,
    recordName: "api",
    target: route53.RecordTarget.fromAlias(new targets.ApiGateway(api)),
});

As I had mentioned previously, this example assumes that you already have a hosted zone defined, and the call above references the hosted zone by the ID. This aliases our API gateway with the api subdomain.

🚀 Deploy and Test

Once you've got everything configured, deploy your stack:

cdk diff
cdk deploy

Your diff should show resources similar to:

Resources
[+] AWS::ApiGateway::RestApi custom-domain-api customdomainapi
[+] AWS::ApiGateway::DomainName custom-domain-api/Domain customdomainapiDomain
[+] AWS::ApiGateway::BasePathMapping custom-domain-api/PathMapping customdomainapiPathMapping
[+] AWS::Route53::RecordSet api-alias-record apializasrecord

After deployment, you should be able to access your API using your custom domain:

curl https://api.yourcompany.com/

You may need to configure an integration in behind the scenes to test this fully end-to-end, and if you do not have an integration already set up, a simple one that you can add is a Lambda integration.

And that's it!

Conclusion

Remember to cdk destroy your stack when you're done testing to avoid unnecessary costs!

Professional URLs: Creating a custom domain for your API Gateway transforms generic AWS URLs into professional, branded endpoints that users can trust and remember.
Flexible Configuration: The combination of domain names, base path mapping, and Route 53 records provides extensive flexibility for organizing multiple APIs under a single domain.
SSL/TLS Security: Built-in HTTPS termination with AWS Certificate Manager ensures your API communications are secure without additional configuration overhead.

How to provision an AWS WAF using AWS CDK

Peter McAree — Thu, 25 Jul 2024 09:35:10 +0000

Security is absolutely paramount to modern applications - especially cloud-based solutions. In fact, it's so critical that specifically within the AWS ecosystem, security is one of the 6 Pillars of the Well-Architected Framework meaning that it should be given an equal amount of thought & effort as everything else that is required for your deployed application.

With that in mind, AWS WAF (Web Application Firewall) is AWS' product that can secure your web applications from common exploits, bots and attacks. It can help mitigate things like DDoS attempts, brute-force attacks and just simply filter your traffic to target your specific audience if you don't necessarily need to have a wide-open public application.

So let's take a look at how we can provision a WAF and attach it to something like an API Gateway - all using AWS CDK.

✍️ Define some CDK

Now unfortunately, AWS CDK does not have any L2 constructs that nicely wrap the AWS WAF functionality, so we'll just create our own reusable components/constructs that wrap the CloudFormation L1 constructs themselves.

Provision the WAF itself

First of all, we want to actually provision the WAF instance itself. Now since I'm going to be attaching this to a REST API Gateway instance (which is a regional instance), this is going to be a regional WAF i.e. region-specific.

Take a look at the following CDK:

import * as waf from "aws-cdk-lib/aws-wafv2";

const defaultActionProperty: waf.CfnWebACL.DefaultActionProperty = {
  block: {
    customResponse: {
      responseCode: 403,
    },
  },
};

const customWaf = new waf.CfnWebACL(this, "api-gateway-waf", {
  defaultAction: defaultActionProperty,
  scope: "REGIONAL",
  name: "api-gateway-waf",
  visibilityConfig: {
    cloudWatchMetricsEnabled: true,
    metricName: "api-gateway-waf",
    sampledRequestsEnabled: true,
  },
});

There are a couple of interesting things here:

We're using the v2 implementation of the WAF service which is the latest & greatest.
Whenever you're provisioning the WAF, you need to specify a defaultAction - this essentially is what action you want to take if the request meets none of the rules. By default, I want to block the request and return a 403 response.
scope is what I briefly mentioned above. It's setting the WAF to either REGIONAL or CLOUDFRONT - pretty self-explanatory, but critical if you're going to attach the WAF to something like an API Gateway or a CloudFront distribution.
Finally, the visibilityConfig is enabling the metrics to be captured and pushed into CloudWatch so that you can analyse and report on the traffic

Create rules for the WAF

Next we want to create and define some rules that we wish to be enforced by the WAF. These are going to define the specifics of which actions you wish to allow/challenge/block etc. There are also managed rules by AWS and customers that are available to choose from - driven by things like OWASP top-10. A note though; depending on which option(s) you choose, it can have different pricing impacts, so always worth checking out the user guide and pricing for more information.

For us, we're just going to define a very basic one to showcase the functionality:

const wafRules: waf.CfnWebACL.RuleProperty[] = [
  {
    name: "GeoLocationRule",
    priority: 0,
    action: {
      allow: {},
    },
    statement: {
      geoMatchStatement: {
        countryCodes: ["US"],
      },
    },
    visibilityConfig: {
      sampledRequestsEnabled: true,
      cloudWatchMetricsEnabled: true,
      metricName: "GeoLocationRule",
    },
  },
];

const customWaf = new waf.CfnWebACL(this, "api-gateway-waf", {
  defaultAction: defaultActionProperty,
  scope: "REGIONAL",
  name: "api-gateway-waf",
  visibilityConfig: {
    cloudWatchMetricsEnabled: true,
    metricName: "api-gateway-waf",
    sampledRequestsEnabled: true,
  },
  rules: wafRules,
});

In the above, you can see that we're just defining a normal array of objects that contain the specific rule configuration. I've defined a GeoLocation rule that allows any traffic origininating from the US. This effectively means that my WAF will only allow traffic from the US - because remember by default action is to block!

Finally, just update the WAF with the rules property and reference our new array.

Attach the WAF to the API Gateway

To attach our WAF to any regional resource that is supported (in our case, an existing API gateway that I have), you need to create a CfnWebACLAssociation construct.

This simply just links the resource and the WAF:

const wafApiGatewayAssociation: waf.CfnWebACLAssociation =
  new waf.CfnWebACLAssociation(this, "waf-api-gateway-association", {
    webAclArn: customWaf.attrArn,
    resourceArn: `arn:aws:apigateway:${cdk.Aws.REGION}::/restapis/${restApiGateway.api.restApiId}/stages/${restApiGateway.api.deploymentStage.stageName}`,
  });

Please note that this is again for the regional resources, for CloudFront distribution(s) associations, you need to configure it separately and won't be able to use this construct.

🚀 Deploy

Whenever you're ready, you can perform a diff & then deploy to check out the resources created and the attachment to your regional resource!

Our diff should look something similar to below:

Resources
[+] AWS::WAFv2::WebACL api-gateway-waf apigatewaywaf
[+] AWS::WAFv2::WebACLAssociation waf-api-gateway-association wafapigatewayassociation

And if you jump into the console, you should see your WAF along with the rule configured (make sure you're pointing at the correct region!):

And you should also see this WAF correctly associated with your resource (in my case, an existing API gateway):

Conclusion

Remember to cdk destroy once you are complete to destroy the CloudFormation stack in order to avoid any unnecessary costs if you're just testing this demo out!

Flexible Security: Provisioning an AWS WAF using AWS CDK enhances web application security effectively.
Custom Components: Utilise CloudFormation L1 constructs to create custom reusable components due to the lack of L2 constructs for AWS WAF in AWS CDK, but since the API is fairly straightforward this is thankfully quite easy!
Traffic Filtering: Define specific rules to filter traffic, allowing you to mitigate threats and control the audience accessing your application.
Regional Attachment: Attach the WAF to regional resources, such as an API Gateway, ensuring appropriate protection for your specific deployment.
Monitoring: Enable metrics and integrate with CloudWatch for valuable insights into traffic patterns and security efficacy.
Best Practices: Adhere to AWS's Well-Architected Framework, ensuring your infrastructure is resilient and secure.
Scalable Security: Maintain robust security settings that can adapt as your application evolves and faces new threats.

How to build a single-page application deployment using AWS CDK

Peter McAree — Sat, 18 May 2024 11:36:00 +0000

All modern applications will require some form of static assets to actually power the functionality for their users, whether it’s media content, supporting scripts or the application content itself.

A super common pattern that I always call upon when deploying single-page applications (SPA) is deploying my static assets into S3 and serving them through CloudFront.

This is incredibly useful when working not only with single-page applications, but also for static assets that need to be served to ensure your applications work as expected.

Let’s take a look at how we can define the infrastructure resources to facilitate serving these assets, using my old favourite AWS CDK.

🔍 Architecture

To give you an idea of what we’re wishing to achieve with this, here is a high-level architecture diagram of what we want to build:

✍️ Let's write some CDK

Crafting the CDK for this pattern is extremely straightforward. There are only the two components:

S3 Bucket
CloudFront Distribution

To provision the S3 bucket, you can use the L2 construct that already exists within CDK:

const s3Bucket = new cdk.aws_s3.Bucket(this, "s3-bucket-content", {
  blockPublicAccess: cdk.aws_s3.BlockPublicAccess.BLOCK_ALL,
  encryption: cdk.aws_s3.BucketEncryption.S3_MANAGED,
  enforceSSL: true,
  removalPolicy: cdk.RemovalPolicy.RETAIN,
  bucketName: "cdk-spa-assets-pattern",
});

With the above, we're just creating a S3 bucket that has public access denied and encryption at rest enabled.

We're then going to create our CloudFront distribution and update the bucket to allow the distribution to read from it:

const originAccessIdentity = new cdk.aws_cloudfront.OriginAccessIdentity(
  this,
  "cloudfront-origin-access-identity"
);
s3Bucket.grantRead(originAccessIdentity);

const cloudfrontS3Origin = new cdk.aws_cloudfront_origins.S3Origin(
  s3Bucket, {
    originAccessIdentity: originAccessIdentity,
  }
);

const distribution = new cdk.aws_cloudfront.Distribution(
  this,
  "cloudfront-content-distribution", {
    defaultBehavior: {
      origin: cloudfrontS3Origin,
    },
    defaultRootObject: "index.html",
    priceClass: cdk.aws_cloudfront.PriceClass.PRICE_CLASS_100,
  }
);

Firstly, we're creating the OriginAccessIdentity construct, this is then used to grant read access to the S3 bucket before assigning it to the CloudFront distribution itself.

We're specifically using the Distribution construct here as it's the newer implementation (replacing the CloudFrontWebDistribution construct). As part of that, the API has changed slightly and we need to define a CloudFront origin itself - we are using S3 as our origin and can pass our origin identity permission into it to allow the distribution to read from the bucket.

We assign that as our default behaviour (as a CloudFront distribution can have multiple behaviours), and finally define our price class.

And that's it! Deploy those and you'll have a S3 bucket that you can push your build assets into. By default, the entry point for the CloudFront distribution is index.html - so whenever you build your SPA assets, you simply push them into the bucket and you will be good to go!

💰 Cost efficiency

Low cost because all of the assets are stored in S3 and served via CloudFront, so your costs will scale with the requests and the traffic coming inbound. Since CloudFront heavily caches at edge, the requests downstream to your S3 bucket will be fairly low - unless you explicitly invalidate the cache if you have new objects.

Conclusion

Static assets can be pushed into S3 bucket and served via the global CloudFront CDN network.
Extremely cost-effective for working with a lot of network requests.
Can work for SPA static websites or other general static content that might be required to power your applications.

Optimising your OpenSearch Ingestion pipeline using AWS CDK

Peter McAree — Mon, 04 Mar 2024 20:54:14 +0000

In my previous post, I delved into how you can use one of the latest features that AWS announced at re:Invent 2023, combining DynamoDB and OpenSearch providing an incredibly powerful zero-ETL ingestion mechanism for driving search and analytic applications.

This integration uses OpenSearch Ingestion Service (OSIS) pipelines under the hood to consume from the DynamoDB streams. The pricing model for the ingestion pipelines is separate from the OpenSearch domains, so you will be charged additionally on top of your instance running.

Depending on your use case, and workloads, you can tweak your ingestion pipelines to optimise for cost - especially for the likes of non-production environments.

Let's take a look at how we might achieve this.

💰 Cost Optimisation

OSIS pipelines are defined in the OpenSearch pricing page under the ingestion section. The charge for majority of regions is $0.24 per OCR per hour - whilst this is a pretty competitive price, depending on your use case, data throughput, how many pipelines you require, and how many environments you are running, the cost can scale up quite quickly.

For example, if you have many pipelines that push into indexes in OpenSearch, some of these may not need their data refreshed as often as others.

An optimisation that can be made, especially for non-production environments, is to only have your pipeline(s) run as often as you need them to (up to a max of 24 hours).

This is made possible by the fact that the DynamoDB stream will retain any data for up to 24 hours and whenever the ingestion pipeline begins, it will consume all of the items on the stream.

Combine this with EventBridge Scheduler, and we have a very powerful optimisation. EventBridge Scheduler allows us to emit StartPipeline and StopPipeline events to target individual pipelines within OSIS. This means that I can have one particular pipeline only run once per day (for an hour) and others running constantly.

✍️ Define some CDK

With the introduction of EventBridge Scheduler, it allows direct integration with thousands of APIs, including the OSIS pipeline APIs.

For my non-production environments, I had no need to run the pipeline for more than once per day, simply because I didn't need the data to be updated and ingested that often.

To run my target pipelines once per day, I just created two separate EventBridge scheduler rules - one for starting and one for stopping.

IAM Role

Firstly, we'll need to create an IAM role that gets assumed by the schedules and has permissions to interface with OSIS:

const eventBridgeOsisIamRole = new cdk.aws_iam.Role(
  this,
  "eventBridgeOsisIamRole", {
    assumedBy: new cdk.aws_iam.ServicePrincipal("scheduler.amazonaws.com"),
    managedPolicies: [
      cdk.aws_iam.ManagedPolicy.fromAwsManagedPolicyName(
        "AmazonOpenSearchIngestionFullAccess",
      ),
    ],
    roleName: "eventBridgeOsisIamRole",
  },
);

EventBridge Schedules

Next we create our two schedules which target the specific OSIS pipeline by the unique name:

const createStartSchedule = (
  pipelineName: string,
  scheduleIamRole: cdk.aws_iam.Role,
) => {
  const CRON_START_PIPELINE_SCHEDULE = "cron(0 5 ? * MON-FRI *)"; // 5am Weekdays
  const startScheduleName = `${pipelineName}-start`;

  return new cdk.aws_scheduler.CfnSchedule(this, startScheduleName, {
    flexibleTimeWindow: {
      mode: "OFF",
    },
    name: startScheduleName,
    scheduleExpression: CRON_START_PIPELINE_SCHEDULE,
    target: {
      arn: "arn:aws:scheduler:::aws-sdk:osis:startPipeline",
      roleArn: scheduleIamRole.roleArn,
      input: JSON.stringify({
        PipelineName: pipelineName,
      }),
    },
  });
};

const createStopSchedule = (
  pipelineName: string,
  scheduleIamRole: cdk.aws_iam.Role,
) => {
  const CRON_STOP_PIPELINE_SCHEDULE = "cron(0 6 ? * MON-FRI *)"; // 6am Weekdays
  const stopScheduleName = `${pipelineName}-stop`;

  return new cdk.aws_scheduler.CfnSchedule(this, stopScheduleName, {
    flexibleTimeWindow: {
      mode: "OFF",
    },
    name: stopScheduleName,
    scheduleExpression: CRON_STOP_PIPELINE_SCHEDULE,
    target: {
      arn: "arn:aws:scheduler:::aws-sdk:osis:stopPipeline",
      roleArn: scheduleIamRole.roleArn,
      input: JSON.stringify({
        PipelineName: pipelineName,
      }),
    },
  });
};

I've abstracted these out to separate methods so that they can be reused in the event that you have multiple pipelines - define the pipeline names and then iterate over them:

["first-osis-pipeline", "second-osis-pipeline"].map((pipelineName: string) => {
  createStartSchedule(pipelineName, eventBridgeOsisIamRole);
  createStopSchedule(pipelineName, eventBridgeOsisIamRole);
});

These constructs will create the two schedules (at 5am & 6am respectively as I've defined in the cron expression), targeting the pipelines and starts/stops using the OSIS API.

And that's it! EventBridge scheduler makes this super easy since it has a first-class integration with the ingestion pipeline API.

Conclusion

Remember to cdk destroy once you are complete to destroy the CloudFormation stack in order to avoid any unnecessary costs if you're just testing this demo out!

Leveraging the fact that DynamoDB streams retain their data change for 24 hours, means that we can optimise our OpenSearch ingestion rate.
This approach works really well for non-production environments and use cases where you don't need to have real-time (or even slightly less frequent) ingestion into OpenSearch.
We can utilise the power of EventBridge Scheduler and the fact that it has a first-class integration with OSIS pipelines to start & stop our pipelines on a cron schedule.
Whenever the pipeline runs and there are items on the DynamoDB stream that haven't been read, it begins consuming and ingesting into OpenSearch.
This allows us to optimise our cost and ensure that we are only consuming OCUs whenever we need to.
Happy optimising!

How to build a zero-ETL DynamoDB integration with OpenSearch Service using AWS CDK

Peter McAree — Sun, 28 Jan 2024 13:18:50 +0000

AWS OpenSearch Service is the managed service offering for deploying search and analytics suites - specifically OpenSearch or Elasticsearch.

At AWS re:Invent 2023, it was announced that people who are using DynamoDB can now ingest their data into OpenSearch, to take advantage of the powerful features that it provides.

The underlying mechanism that provides this functionality is Amazon OpenSearch Ingestion in combination with S3 exports and DynamoDB streams. It essentially allows developers to push data into DynamoDB tables as normal and this data will be automatically ingested (and updated!) into OpenSearch.

As part of the pipeline, it can also be separately backed up to S3 as well which is useful for re-ingesting in the future if required.

This is an extremely powerful coupling of services, and a cost-effective way to ingest data into OpenSearch. Let's take a look at how we can achieve this using AWS CDK.

✍️ Define some CDK

There are a few things that we're going to need to provision in order to set up our ingestion pipeline - although I'll assume that you have an existing OpenSearch Domain running.

DynamoDB Table

Let's spin up our DynamoDB table that we'll push items in to.

const dynamoDbTable = new cdk.aws_dynamodb.TableV2(this, "DynamoDB Table", {
  partitionKey: {
    name: "id",
    type: cdk.aws_dynamodb.AttributeType.STRING
  },
  sortKey: {
    name: "timestamp",
    type: cdk.aws_dynamodb.AttributeType.NUMBER,
  },
  tableName: "opensearch-ingestion-table",
  billing: cdk.aws_dynamodb.Billing.onDemand(),
  pointInTimeRecovery: true,
  dynamoStream: cdk.aws_dynamodb.StreamViewType.NEW_IMAGE,
});

Some things to note here:

We need to ensure that Point in time recovery (PITR) is enabled, as this is required for the pipeline integration.
DynamoDB streams also need to be enabled to capture any changes to items that also get subsequently ingested into OpenSearch.

S3 Bucket and IAM Role

Next up is the S3 bucket that we'll use to backup the raw events that get consumed by the OpenSearch pipeline, and the IAM role that will be assumed by the pipeline:

const s3BackupBucket = new cdk.aws_s3.Bucket(
  this,
  "OpenSearch DynamoDB Ingestion Backup S3 Bucket", {
    blockPublicAccess: cdk.aws_s3.BlockPublicAccess.BLOCK_ALL,
    bucketName: "opensearch-ddb-ingestion-backup",
    encryption: cdk.aws_s3.BucketEncryption.S3_MANAGED,
    enforceSSL: true,
    versioned: true,
    removalPolicy: cdk.RemovalPolicy.RETAIN,
  }
);

const openSearchIntegrationPipelineIamRole = new cdk.aws_iam.Role(
  this,
  "OpenSearch Ingestion Pipeline Role - DynamoDB", {
    roleName: "OpenSearch Ingestion Pipeline Role - DynamoDB",
    assumedBy: new cdk.aws_iam.ServicePrincipal(
      "osis-pipelines.amazonaws.com"
    ),
    inlinePolicies: {
      openSearchIntegrationPipelineIamRole: new cdk.aws_iam.PolicyDocument({
        statements: [
          new cdk.aws_iam.PolicyStatement({
            actions: ["es:DescribeDomain"],
            resources: [
              `arn:aws:es:eu-west-1:*:domain/${openSearchDomain.domainName}`,
            ],
            effect: cdk.aws_iam.Effect.ALLOW,
          }),
          new cdk.aws_iam.PolicyStatement({
            actions: ["es:ESHttp*"],
            resources: [
              `arn:aws:es:eu-west-1:*:domain/${openSearchDomain.domainName}`,
            ],
            effect: cdk.aws_iam.Effect.ALLOW,
          }),
        ],
      }),
    },
  }
);

openSearchIntegrationPipelineIamRole.addManagedPolicy(
  cdk.aws_iam.ManagedPolicy.fromAwsManagedPolicyName(
    "AmazonDynamoDBFullAccess"
  )
);
openSearchIntegrationPipelineIamRole.addManagedPolicy(
  cdk.aws_iam.ManagedPolicy.fromAwsManagedPolicyName("AmazonS3FullAccess")
);

When it comes to the S3 bucket, it's fairly standard - no public access, encrypted at rest and versioned.

With the IAM Role, we need to allow the role to be assumed by the OpenSearch Ingestion Service (OSIS) pipelines. We then provide it with some specific OpenSearch Service permissions, before adding DynamoDB and S3 access - these could be tailored better to adhere to the principle of least privilege but for ease of showcasing the functionality, I've just defaulted to the managed full access policies.

OpenSearch Service Pipeline

Finally, we need to define the pipeline construct and the configuration for said pipeline.

The configuration for the pipeline is a data-prepper feature of OpenSearch and the specific documentation for DynamoDB (and the API) can be found here - take a look at the other sources and sinks that are possible out of the box.

Now you'll probably notice that the config is defined as YAML, but since we're defining it as infrastructure as code using TypeScript and want some dynamic variables in there, I just wrapped it up in a separate function that just interpolates strings.

const generateTemplate = (
  dynamoDbTable: cdk.aws_dynamodb.TableV2,
  s3BucketName: string,
  iamRoleArn: string,
  openSearchHost: string,
  indexName: string,
  indexMapping: string
) => `
version: "2"
dynamodb-pipeline:
  source:
    dynamodb:
      acknowledgments: true
      tables:
        - table_arn: "${dynamoDbTable.tableArn}"
          # Remove the stream block if only export is needed
          stream:
            start_position: "LATEST"
          # Remove the export block if only stream is needed
          export:
            s3_bucket: "${s3BucketName}"
            s3_region: "eu-west-1"
            s3_prefix: "${dynamoDbTable.tableName}/"
      aws:
        sts_role_arn: "${iamRoleArn}"
        region: "eu-west-1"
  sink:
    - opensearch:
        hosts:
          [
            "https://${openSearchHost}",
          ]
        index: "${indexName}"
        index_type: "custom"
        template_content: |
          ${indexMapping}
        document_id: '\${getMetadata("primary_key")}'
        action: '\${getMetadata("opensearch_action")}'
        document_version: '\${getMetadata("document_version")}'
        document_version_type: "external"
        bulk_size: 4
        aws:
          sts_role_arn: "${iamRoleArn}"
          region: "eu-west-1"
`;

This really defines what we want our ingestion pipeline to do. This is pretty much the example config that is in the documentation, with a few critical tweaks.

For the source portion of the config, we're ultimately:

Defining that DynamoDB is our source, which table we want to ingest and the position of the stream to start from.
As well as ingesting the stream into OpenSearch, we also want to export to S3 as a form of a backup, so we define the target bucket.
Finally, we're setting the IAM role that we want the ingestion pipeline to use. Note: the documentation is specific about what permissions and policies need to be attached to it, so make sure to reference them!

For the sink configuration:

Pointing it to our OpenSearch domain cluster via setting the host.
Specifying the index name, what type it is and critically, we're also setting template_content which is essentially the index mapping - but more on this below.
Setting various document related metadata which are utilising internal intrinsic functions here that are unique to the DynamoDB integration along with the maximum bulk size of requests to be sent to OpenSearch in MB.
Then again, just specifying the IAM role for the sink portion of the pipeline to use.

Something that is incredible powerful when using OpenSearch is if you know the structure of the data you're going to be ingesting ahead of time, meaning you can define the index template (or mapping) - specifically setting the data types for each field of the document.

In this case, we're defining the template_content which is exactly that in a JSON-representation.

const indexName = "log-events";
const indexMapping = {
  settings: {
    number_of_shards: 1,
    number_of_replicas: 0,
  },
  mappings: {
    properties: {
      id: {
        type: "keyword",
      },
      timestamp: {
        type: "date",
        format: "epoch_millis",
      },
    },
  },
};

const pipelineConfigurationBody = generateTemplate(
  dynamoDbTable,
  backupS3Bucket.bucketName,
  openSearchPipelineRole.roleArn,
  openSearchDomain.domainEndpoint,
  indexName,
  JSON.stringify(indexMapping)
);

For the purposes of a demo, I've just kept this fairly trivial with only two properties - but you can see how it can be easily extended in this case.

Finally, we bring all of this together by creating the OSIS pipeline resource itself. There isn't a L2 CDK construct for this, so we're just using the lower-level CloudFormation pipeline construct, but it works perfectly:

const cloudwatchLogsGroup = new cdk.aws_logs.LogGroup(
  scope,
  "OpenSearch DynamoDB Ingestion Pipeline Log Group", {
    logGroupName: `/aws/vendedlogs/OpenSearchIntegration/opensearch-dynamodb-ingestion-pipeline`,
    retention: cdk.aws_logs.RetentionDays.ONE_MONTH,
  }
);

const cfnPipeline = new cdk.aws_osis.CfnPipeline(
  scope,
  "OpenSearch DynamoDB Ingestion Pipeline Configuration", {
    maxUnits: 4,
    minUnits: 1,
    pipelineConfigurationBody: pipelineConfigurationBody,
    pipelineName: "dynamodb-integration",
    logPublishingOptions: {
      cloudWatchLogDestination: {
        logGroup: cloudwatchLogsGroup.logGroupName,
      },
      isLoggingEnabled: true,
    },
  }
);

As straightforward as defining our OpenSearch Compute Units (OCUs) boundaries that we want the pipeline to work within (these are the billable components within the ingestion pipelines), and setting the pipeline configuration that we've generated along with some additional CloudWatch logging for debugging and monitoring.

And now we're ready to deploy!

🚀 Deploy!

Let's deploy this example and see what it looks like in practice. Our CDK diff should resemble something like the following:

Resources
[+] AWS::DynamoDB::GlobalTable DynamoDB Table DynamoDBTable8EA388EE
[+] AWS::IAM::Role OpenSearch Ingestion Pipeline Role - DynamoDB OpenSearchIngestionPipelineRoleDynamoDBC974378B
[+] AWS::S3::Bucket OpenSearch DynamoDB Ingestion Backup S3 Bucket OpenSearchDynamoDBIngestionBackupS3Bucket017A2C31
[+] AWS::S3::BucketPolicy OpenSearch DynamoDB Ingestion Backup S3 Bucket/Policy OpenSearchDynamoDBIngestionBackupS3BucketPolicy4CA396B4
[+] AWS::Logs::LogGroup OpenSearch DynamoDB Ingestion Pipeline Log Group OpenSearchDynamoDBIngestionPipelineLogGroupEB6F0DB2
[+] AWS::OSIS::Pipeline OpenSearch DynamoDB Ingestion Pipeline Configuration OpenSearchDynamoDBIngestionPipelineConfiguration

Once the resources have deployed, we can test our ingestion by adding an item into the DynamoDB table that has just been provisioned - but let's check the pipeline has been configured as expected.

Jump into the console and firstly into OpenSearch, navigate to the pipelines section and you should see your pipeline along with the configuration we've set up:

Can also verify this from the DynamoDB end of things, via the Integrations section of the tables as well:

And that's it! You can now put items into your DynamoDB table and they should be ingested.

Note: if you don't see your index or any items ingested, jump into CloudWatch logs for the pipeline. If you're pointing to an existing OpenSearch cluster, make sure to check the permissions and fine-grained access control are sufficient.

Conclusion

Remember to cdk destroy once you are complete to destroy the CloudFormation stack in order to avoid any unnecessary costs if you're just testing this demo out!

Combining the ease of the interface with DynamoDB with an ingestion pipeline into OpenSearch is incredibly powerful. If you're already utilising DynamoDB, your application logic doesn't need to change - it just needs to continue to write items to it as normal.
The advantage of the OpenSearch Ingestion Service pipeline is that it can consume from your DynamoDB stream and also export to a S3 bucket too in order to keep a backup of your events/items in the case that you need to re-index in OpenSearch.
You can specify how you wish to set the index up in OpenSearch in the pipeline itself, meaning you can have fine-grained control over the number of shards, replicas and the mapping structure - ensuring that the integration with DDB is optimised.
It's a cost-effective solution for users who don't need to perform any additional transformation as part of their ingestion pipeline, i.e. you don't necessarily need to use Kinesis (and it being quite costly!).
Some cost optimisation can be applied to the pipelines in order to keep costs as low as possible depending on your use case, but that will come in a follow-up blog!
Whilst there aren't explicit L2 CDK constructs for some of the resources, the API is fairly straightforward to utilise.
Be sure to check out the announcement along with some of the other linked documentation in there for the OSIS pipeline and DynamoDB table setup - happy searching!

Drive to PartyRock

Peter McAree — Fri, 17 Nov 2023 09:12:00 +0000

AWS released a new playground yesterday (16th November) dubbed PartyRock. PartyRock is an Amazon Bedrock playground that allows you to build small, but powerful generative AI applications based on a natural language prompt.

Consuming the prompt, it will create widgets based on the various outcomes that you wish your application to produce. This is super exciting because it opens the door to incredibly lightweight and useful interfaces that everyone can build - lowering the barrier of entry to GenAI even further.

I got preview access to PartyRock through the AWS Community Builders program and decided to test it out with a fun, and somewhat time-relevant application 😄

🏎 Accelerating to the finish line

Being an avid Formula 1 fan, with the Las Vegas Grand Prix taking place this weekend and AWS re:Invent just round the corner also taking place in Vegas, it only really meant one thing that I could test PartyRock out with.

Enter the F1 Race Guide: Your Ultimate Trackside and Off-Track Planner.

I thought it would be cool to build an application that would consume a single country as a prompt and see what the various AI models in behind the scenes could muster together for some key information if you were ever planning a trip to a F1 race. Some of the key researching points before you go include:

What are the best seats
The range of ticket prices
Best value for money vs. viewing point
Non-race activities and general tourist/sightseeing activities

So, I thought I'd put PartyRock to the test.

🤖 It all starts with a brief prompt

Like I mentioned above, getting started with PartyRock has an incredibly low barrier to entry - it literally consumes a single natural language prompt to generate your entire application. As with any GenAI application, it's important to be clear and concise about what you want your application to do and the outcomes you wish to achieve. So I started with the following prompt:

I want the app to be the ultimate Formula 1 companion when it comes to planning and researching attending a race. The app should consume the location the user wishes to research and suggest where the best seats are to sit, the ticket prices, rating of the view point of the seats suggested and a generated image of what the viewpoint might look like. On top of this, it should also generate some general tourist activities that the user can engage with.

Within the minute, PartyRock had the created this nice app on my behalf which included a few widgets based on the prompt.

For the purposes of testing this out, I didn't fine tune any of the prompts or parameters for each of the widgets - I simply just let the model roll with what it initially generated, but more on the fine tuning later!

♠️ Roll the dice

First of all, let's test it out with the Grand Prix taking place this weekend - Las Vegas! And like I said, with AWS re:Invent only round the corner, it just feels right.

The app generated the following output as a suggestion for the "Best Seats":

I absolutely love some of the suggestions including grandstand tickets to general admission, as in some cases depending on the track and what view you're looking for, general admission can be optimal!

This sounds pretty cool - but with Las Vegas being a new race to the F1 calendar and the track layout changing a few times since its initial announcement, I was curious to cross-reference some of its suggestions with what the confirmed layout was.

Turn 1 grandstands with some hard braking, always great action especially from the start - this checks out! Whilst there is a chicane section of the track, it's more turns 7-8-9 now with the changes, but close enough and good general advice for some guaranteed action. It also does mention an elusive turn 18, which doesn't quite exist as we only have 17, so maybe not the best track to start with given it's brand new and LLM are traditionally slightly outdated.

Let's take a look at some of the other outputs from the app. For view rating, it doesn't have enough information from each grandstand location to reliably rate which again, is fair given that it's a new track and we also haven't been specific with what grandstand we want to sit in.

Ticket price is a similar story, however it does give some general guidelines and rough estimates which is always welcoming when planning a trip to any race!

Our view image is pretty cool - like an abstract piece of art, almost like a promotional race poster? You've got the grandstands, barriers, sweeping flow of traffic - maybe not entirely accurate of what a viewpoint on a particular grandstand would look like, but I think it's pretty cool nonetheless.

And finally, the tourist suggestions output gave plenty of suggestions for some of the many touristy things to do whilst in Vegas:

All extremely viable options for non-race days, apart from the Bellagio Fountains as it might be quite hard to see them this weekend with all of the race infrastructure in place. Unless you are lucky enough (and have deep enough pockets) for the Bellagio Fountain Club 😎

Overall, pretty cool output - some good suggestions and understandable that some of the information will have been outdated since the release of the track and the changing of the layouts. However, pretty impressed that it was even able to output anything at all given all of those facts above!

🪨 How about something silver?

Let's give it another go but this time, with a more traditional race that has featured on the F1 calendar over the years with a more defined layout - Silverstone.

For best seats, it gave me a good few solid options including some F1 fan favourites which absolutely checks out (in my opinion!).

If we quickly cross-reference these suggestions with the actual track & grandstand layout, you'll notice it's using the specific turn names as that is what they are typically called. Becketts grandstand is in fact, one of the best for Silverstone and the budget option of general admission at Club Corner is accurate as well - although you'll need to entering the track bright and early to get a good view if that's the route you're going!

For view rating, it was a similar story as Vegas - we haven't been super specific with what grandstand or area we want to sit, so it can't accurately suggest ratings for the viewpoint.

Ticket prices, again, was similar - not specific enough so can't be totally accurate, however it did give us general price ranges for a lot of different options including advanced purchasing discount which is pretty cool!

The image it generated you could almost argue looks like it's the final turn out of the Becketts complex heading down the Hangar straight? Pretty cool piece of generated art though:

And finally, the tourist options for non-race days were as follows:

I would have liked to maybe have seen some suggests for F1 team factory tours if they were available given the context of the app and the fact that there are quite a few teams based not too far from the Silverstone circuit between Brackley and Milton Keynes - but still some solid suggestions!

⚡️ Improving

PartyRock sits on top of AWS Bedrock, so it provides various models to choose from to tailor to your needs and also allows some customisation of how it might react. For example, it will allow you to customise the prompt for each widget that it generates, along with updating the temperature and top-p helping you tailor it to your specific needs.

Conclusion

PartyRock is a pretty awesome, lightweight tool that will allow people to experiment and experience GenAI with a light-hearted touch but also providing insights into what the possibilities with it are.

In this first iteration of PartyRock, it doesn't have internet access enabled, so that might explain why some of the details weren't as accurate as they could be. However, I still think given that I deliberately didn't fine tune anything outside of the initial prompt for testing purposes, it gave pretty solid outcomes that I could actually come away and possibly research further or investigate myself.

The little application that I (or PartyRock I suppose!) crafted is publicly available and you can test it out for yourself! You can even remix it and tweak the prompts yourself to see what other applications you can get it to produce.

Test it out here: https://partyrock.aws/u/pmc-a/l0oODwl7_/F1-Race-Guide%3A-Your-Ultimate-Trackside-and-Off-Track-Planner and please comment/let me know what outcomes and testing you perform with it!

Head on over to AWS PartyRock for more details - it's free to signup and use for a limited time, so why not give it a try?

How to build AWS State Machines using AWS CDK - Part III

Peter McAree — Wed, 08 Nov 2023 13:00:11 +0000

This is the third-part in a blog mini-series showcasing the functionality of AWS Step Function state machines and how to express them as infrastructure as code using AWS CDK. If you haven’t read the first two parts, check them out here and here — as we continue using the same example for continuity between each.

Last time, we focused on the choice flow operation — this allows us to perform conditional evaluations based on different state parameters that are available (and potentially mutated) throughout your state machine.

During this blog, we’re going to focus on some of the other flow operations that are available to you when defining your state machine. These operations can optimise and improve the processing of your data.

❓ What

As I mentioned above, we’re going to focus on the remaining flow operations that are available to you when crafting your state machine. A reminder of all of the flow operators available at your disposal:

Choice (if/then-else logic)
Parallel
Map
Wait

Continuing on with building up the same state machine, we have already interfaced with the choice and wait operations, so within this one, we’re going to hone in on the parallel and map operations — how they can be defined and used.

So let’s dive in and see some potential use cases for these operations!

✍️ Define some CDK

Picking up from where we left off last time, our final state machine CDK definition looked something like the following:

import * as cdk from "aws-cdk-lib";  
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";  
import * as lambda from "aws-cdk-lib/aws-lambda";  

import * as stepFunctions from "aws-cdk-lib/aws-stepfunctions";  
import * as stepFunctionsTasks from "aws-cdk-lib/aws-stepfunctions-tasks";  

const mockLambdaFunctionArn =  
  "arn:aws:lambda:us-east-1:12345:function:my-shiny-lambda-function";  
const mockDdbTableArn =  
  "arn:aws:dynamodb:us-east-1:12345:table/my-shiny-dynamodb-table";  

const lambdaFunction = lambda.Function.fromFunctionArn(  
  this,  
  "lambda-function",  
  mockLambdaFunctionArn  
);  
const dynamodbTable = dynamodb.Table.fromTableArn(  
  this,  
  "dynamo-db-table",  
  mockDdbTableArn  
);  

const processJob = new stepFunctionsTasks.LambdaInvoke(  
  this,  
  "state-machine-process-job-fn", {  
    lambdaFunction: lambdaFunction,  
    resultPath: "$.processJobResult",  
  }  
);  

const wait10MinsTask = new stepFunctions.Wait(  
  this,  
  "state-machine-wait-job", {  
    time: stepFunctions.WaitTime.duration(cdk.Duration.minutes(10)),  
  }  
);  

const ddbWrite = new stepFunctionsTasks.DynamoPutItem(  
  this,  
  "ddb-write-job", {  
    item: {  
      uuid: stepFunctionsTasks.DynamoAttributeValue.fromString(  
        crypto.randomUUID()  
      ),  
      timestamp: stepFunctionsTasks.DynamoAttributeValue.fromString(  
        new Date().toISOString()  
      ),  
    },  
    table: dynamodbTable,  
    resultPath: "$.ddbWriteResult",  
  }  
);  

// Constructs for if/else choice block  
const conditionalMatchLambdaFunction = new stepFunctionsTasks.LambdaInvoke(  
  this,  
  "state-machine-conditional-match-fn",  
  {  
    lambdaFunction: lambdaFunction,  
  }  
);  
const choiceStatement = new stepFunctions.Choice(  
  this,  
  "conditional-choice-block"  
);  
const choiceCondition = stepFunctions.Condition.booleanEquals(  
  "$.pass",  
  true  
);  
const elsePassStep = new stepFunctions.Pass(this, "else-block-pass");  

const stateMachineDefinition = processJob  
  .next(wait10MinsTask)  
  .next(ddbWrite)  
  .next(  
    choiceStatement  
      .when(choiceCondition, conditionalMatchLambdaFunction)  
      .otherwise(elsePassStep)  
  );  

const stateMachine = new stepFunctions.StateMachine(this, "state-machine", {  
  definitionBody: stepFunctions.DefinitionBody.fromChainable(  
    stateMachineDefinition  
  ),  
  timeout: cdk.Duration.minutes(15),  
  stateMachineName: "ProcessAndReportJob",  
});

⚡️ Parallel

The parallel flow operation can be used to add separate branches to your definition in order to facilitate parallelisation in your state machine. Critically though, it will wait until all branches defined in your parallel state reach a terminal state before proceeding onto the next block.

The branches in the parallel state don’t need to perform the same operation, nor do they need to produce the same structured output.

Let’s take a look at how we can define this within CDK:

import * as cdk from "aws-cdk-lib";  
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";  
import * as lambda from "aws-cdk-lib/aws-lambda";  
import * as stepFunctions from "aws-cdk-lib/aws-stepfunctions";  

// Constructs for parallel block  
const addressLookupLambda = new stepFunctionsTasks.LambdaInvoke(  
  this,  
  "address-lookup-fn", {  
    lambdaFunction: lambdaFunction,  
    resultPath: "$.addressLookupLambdaResult",  
  }  
);  

const phoneLookupLambda = new stepFunctionsTasks.LambdaInvoke(  
  this,  
  "phone-lookup-fn", {  
    lambdaFunction: lambdaFunction,  
    resultPath: "$.phoneLookupLambdaResult",  
  }  
);  

const parallelBlock = new stepFunctions.Parallel(  
  this,  
  "PII Parallel Process"  
)  
 .branch(addressLookupLambda)  
 .branch(phoneLookupLambda);

So we’re using a fairly trivial example here to showcase the syntax and setup, but you can see that it’s straightforward to define a parallel process using the construct available.

Here we are just defining two lambda function tasks, which we have previously defined elsewhere in the CDK, that are going to perform two separate tasks/lookups to fetch some data.

We then specify our Parallel block which contains two chainable methods using .branch() and these are what determine which tasks to perform in parallel — so in our case, it’s both of our lambda invocations. Like I said above, these don’t have to be the same task performed, reading from the same resources etc. Any of the plethora of state machine tasks can be invoked here, just critical to remember that the parallel block will wait until all tasks have completed before proceeding any further in the state machine.

Note: whilst you can have more than 2 branches in parallel and there doesn’t appear to be an explicit upper limit within the AWS documentation, I’d recommend not invoking more than 5 processes in parallel. If you have a larger orchestration platform with multiple large async running processes, it might be worth considering separate patterns.

Finally, we just append the parallelBlock onto our state machine definition like so:

const stateMachineDefinition = processJob  
  .next(wait10MinsTask)  
  .next(ddbWrite)  
  .next(parallelBlock)  
  .next(  
    choiceStatement  
      .when(choiceCondition, conditionalMatchLambdaFunction)  
      .otherwise(elsePassStep)  
  );

And with some valid execution input, the state machine will execute as expected and run the two lookup lambdas in parallel — resulting in our graph looking like the following:

When it comes to the output of the parallel block, since it waits for all tasks to complete, the output from a parallel block is an array of objects. Depending on what resultPath or outputPath you configure for each of the task blocks within the parallel operation, it will return an array containing the output from every task.

So in our case above, our output from the parallel block looks similar to the following structure:

{  
  "pass": true,  
  "processJobResult": { … },  
  "ddbWriteResult": { … },  
  "parallelBlockOutput": [  
    {  
      "addressLookupLambdaResult": { … }  
    },  
    {  
      "phoneLookupLambdaResult": { … }  
    }  
  ]  
}

This can then be parsed and passed onto the next state transition in the state machine for conditional logic, further processing or storage elsewhere.

🗺 Map

The final major flow operation that you can utilise is map. This operation allows you to process items in a data set concurrently within your state machine.

There are two options for processing the items within the map:

Inline
Distributed

The main difference between these two is how much data can be processed. Inline is useful for smaller data sets and you don’t require large amounts of concurrency as inline only supports a maximum of 40 concurrent iterations.

In contrast, distributed is the high-concurrency mode and can process up to 10,000 concurrent iterations — however, it does come with some other separate limitations to consider before using.

However, for most use cases within a single state machine execution, both of these options should be more than sufficient unless you are dealing with extremely high-volume and high throughput ETL processes.

So let’s have a look at how we might define this in CDK:

const mapBlock = new stepFunctions.Map(this, "Transform Map Process", {  
  maxConcurrency: 5,  
  itemsPath: "$.itemsToTransform",  
  resultPath: "$.mapBlockOutput",  
});

We first start by defining the Map construct itself. This defines the core properties that the iterator will execute the task against. In the above, we’re explicitly setting the maxConcurrency, specifying the itemsPath (where the array is defined in your input via JSONpath for map to iterate over) and finally the resultPath which defines where the output of the map tasks are to be output to.

We then need to define the task/process we want to perform within the map iterator itself:

const transformIteratorFn = new stepFunctionsTasks.LambdaInvoke(  
  this,  
  "transform-iterator-fn", {  
    lambdaFunction: lambdaFunction,  
  }  
);

Again, for showcase purposes, I’ve just reused the same Lambda function task definition that we have used for most of our tasks — however, this can be swapped out with any chainable task methods that can be defined in the state machine. These can be useful for almost separate sub-routine definitions that you want to perform inline inside your overall state machine.

So now, for each item in our array, we are going to invoke this lambda function before finally outputting the result to our defined resultPath.

Note: in a similar fashion to the parallel operation, the state machine will not proceed onto the next step in your state machine until all of the iterations within the map have completed.

Finally, we just append this map construct onto our overall state machine definition:

const stateMachineDefinition = processJob  
  .next(wait10MinsTask)  
  .next(ddbWrite)  
  .next(parallelBlock)  
  .next(mapBlock.iterator(transformIteratorFn))  
  .next(  
    choiceStatement  
    .when(choiceCondition, conditionalMatchLambdaFunction)  
    .otherwise(elsePassStep)  
  );

Once deployed, we can define some input criteria for an execution of our state machine:

{  
  "pass": true,  
  "itemsToTransform": ["Joe", "Rachael", "Steve", "Jenny"]  
}

And the lovely output of our execution should look something like this!

There we have it — a state machine that contains a map operation that can be used for concurrent processing of data, nice.

Conclusion

Remember to cdk destroy once you’re done to tear down the Cloudformation stack!

Flow operations within state machines are incredibly useful to create advanced ETL processes, event-driven orchestration and just general input/output management between different processes.
Parallel operations allow you to perform different tasks concurrently before returning the result of the branches for further computation, thus speeding up the time your state machine takes to complete.
Map operations allow you to iterate over a list of items and perform the same task on them concurrently, speeding up the time of completion.
I’ve only really scratched the surface with some trivial examples here of what is possible and display the CDK interfaces for defining these constructs. I would encourage you to explore the documentation more to find out the other features of these operations that could be of use to you.
You can view the additional API properties for the parallel operation here and for the map operation, you can read more about the concepts of how it works and some of the limitations that I mentioned above (inline vs. distributed) that can assist you with picking the best option for your use case.

How to build AWS State Machines using AWS CDK - Part II

Peter McAree — Fri, 22 Sep 2023 08:45:51 +0000

In my previous blog in this mini-series, we walked through what step functions are used for, how they can be defined and then continued to define a basic one with a few building blocks in CDK. If you haven't read it, check it out here.

Within this blog, we're going to take a look at some of the other constructs and concepts that you can utilise within your state machine for optimisation and power-processing of data. We're going to build on top of the existing state machine that we created last time and focus on the choice logic operator.

❓ What

As I described before, Step Function state machines have a plethora of low-level AWS service integrations that you can interface with directly - these are typically known as actions. They also have a bunch of different flow operations that you can apply to your state machine that allows finer control of what happens to the data that you're passing in.

Examples of flow operations include:

Choice (if/then-else logic)
Parallel
Map
Wait

In the existing state machine that we're going to build on top of for these examples, you might have noticed that we're already defining a wait flow operation - so let's enhance and add a few more for a prescriptive example.

✍️ Define some CDK

Picking up from where we left off last time, our final state machine CDK definition looked something like the following:

import * as cdk from "aws-cdk-lib";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";
import * as lambda from "aws-cdk-lib/aws-lambda";

import * as stepFunctions from "aws-cdk-lib/aws-stepfunctions";
import * as stepFunctionsTasks from "aws-cdk-lib/aws-stepfunctions-tasks";

const mockLambdaFunctionArn =
  "arn:aws:lambda:us-east-1:12345:function:my-shiny-lambda-function";
const mockDdbTableArn =
  "arn:aws:dynamodb:us-east-1:12345:table/my-shiny-dynamodb-table";

const lambdaFunction = lambda.Function.fromFunctionArn(
  this,
  "lambda-function",
  mockLambdaFunctionArn
);
const dynamodbTable = dynamodb.Table.fromTableArn(
  this,
  "dynamo-db-table",
  mockDdbTableArn
);

const processJob = new stepFunctionsTasks.LambdaInvoke(
  this,
  "state-machine-process-job-fn", {
    lambdaFunction: lambdaFunction,
  }
);

const wait10MinsTask = new stepFunctions.Wait(
  this,
  "state-machine-wait-job", {
    time: stepFunctions.WaitTime.duration(cdk.Duration.minutes(10)),
  }
);

const ddbWrite = new stepFunctionsTasks.DynamoPutItem(
  this,
  "ddb-write-job", {
    item: {
      uuid: stepFunctionsTasks.DynamoAttributeValue.fromString(
        crypto.randomUUID()
      ),
      timestamp: stepFunctionsTasks.DynamoAttributeValue.fromString(
        new Date().toISOString()
      ),
    },
    table: dynamodbTable,
  }
);

const stateMachineDefinition = processJob
  .next(wait10MinsTask)
  .next(ddbWrite);

const stateMachine = new stepFunctions.StateMachine(this, "state-machine", {
  definitionBody: stepFunctions.DefinitionBody.fromChainable(
    stateMachineDefinition
  ),
  timeout: cdk.Duration.minutes(5),
  stateMachineName: "ProcessAndReportJob",
});

So you can see from the above, within the CDK library, there is a specific aws-stepfunctions-tasks namespace that houses a lot of the actions that can be performed within a task block in state machines. These include things like Lambda invocations, DynamoDB put item operations, SQS enqueue item etc.

For the flow operations, we will be using the aws-stepfunctions namespace to perform some of this logical control that we want to achieve.

🔍 Choice

A choice flow operation is pretty much exactly as it is described. It's an if/else statement that can apply conditional branching logic to your state machine in order to perform different actions depending on the input data that has been passed in.

Critically, this input data can either be at the top-level of the state machine (i.e. the state machine execution input) or it can be derived from other metadata that has been output from other steps (e.g. a response from a Lambda function invocation).

To keep this fairly simple, we're going to just focus on the top-level execution input data and for demonstration purposes, it's going to be a boolean value.

Firstly, we're going to create our choice block construct along with the condition we want to evaluate:

const choiceStatement = new stepFunctions.Choice(
  this,
  "conditional-choice-block"
);
const choiceCondition = stepFunctions.Condition.booleanEquals(
  "$.pass",
  true
);

You can see in the snippet, the value we want to evaluate is pass but it has some slightly odd looking syntax. This is JSON path notation. Inputs & outputs to state machines and the individual workflows inside the state machine all flow using JSON.

JSON path is how we can reference different variables and mutate the inputs/outputs if we need to. You can read more about how inputs/outputs work within state machines here.

AWS also have some tooling available for learning & testing the various paths, parameters etc. within state machines which I will link at the end!

This is a pretty important concept to understand, as it is used heavily throughout any implementation of state machines you might have for managing state transitions! You'll even notice in one of the upcoming snippets that we have to modify existing action operations within our state machine to cater for how these inputs/outputs work.

Continuing on! Next, we need to define what actions we want to perform in each of our conditional branches - for our if branch, we're going to invoke a Lambda function and in the else branch, we're just going to have a Pass state. Let's set this up:

const conditionalMatchLambdaFunction = new stepFunctionsTasks.LambdaInvoke(
  this,
  "state-machine-conditional-match-fn", {
    lambdaFunction: lambdaFunction,
  }
);

const elsePassStep = new stepFunctions.Pass(this, "else-block-pass");

Finally, we need to update our state machine definition with all of these references we have just created using the familiar chaining methods you might have seen before:

const stateMachineDefinition = processJob
  .next(wait10MinsTask)
  .next(ddbWrite)
  .next(
    choiceStatement
    .when(choiceCondition, conditionalMatchLambdaFunction)
    .otherwise(elsePassStep)
  );

Specifically focusing on our choice implementation here, you can see that we are chaining two methods together to perform the conditional logic

when defines our if clause and action we want to perform
otherwise defines our else clause and operation we want to perform

A fairly clean API in my opinion! BUT WAIT.

Remember I had mentioned above about JSON path and how important it was to understand for inputs/outputs?

Well if you were to deploy and create an execution of your state machine at the moment with the following execution input:

{
  "pass": true
}

You will actually get a runtime error describing that the choice block cannot evaluate the path you've provided of $.pass due to it not being present within the input.

This is due to managing state between your workflow transitions. $.pass doesn't exist on the input because of a few issues:

The state machine has processed a number of different actions and flow operations prior to our choice
We aren't actually specifying what we want to do with the output of each of these actions and flow operations.

By default, the input to the next workflow item in your state machine will be the entire output of the previous workflow item. In our case, this means that the input to our choice flow operation will be the output from the DynamoDB PutItem action!

In order to solve this, we can make a small modification to the two previous action workflow items so that they don't override the initial execution input, but rather append onto it:

const processJob = new stepFunctionsTasks.LambdaInvoke(
  this,
  "state-machine-process-job-fn", {
    lambdaFunction: lambdaFunction,
    resultPath: "$.processJobResult",
  }
);

const ddbWrite = new stepFunctionsTasks.DynamoPutItem(
  this,
  "ddb-write-job", {
    item: {
      uuid: stepFunctionsTasks.DynamoAttributeValue.fromString(
        crypto.randomUUID()
      ),
      timestamp: stepFunctionsTasks.DynamoAttributeValue.fromString(
        new Date().toISOString()
      ),
    },
    table: dynamodbTable,
    resultPath: "$.ddbWriteResult",
  }
);

The critical change here is adding the resultPath property onto the object for both our Lambda function invocation and our DynamoDB PutItem operation. This resultPath specifies where the output of the action block in question should be appended on to - in this case, I've just named them similarly to the variables themselves.

However with this change, the input to our choice block now looks like:

{
  "pass": true,
  "processJobResult": {
    "ExecutedVersion": "$LATEST",
    "Payload": { ... },
    ...
    "StatusCode": 200
  },
  "ddbWriteResult": {
    "SdkHttpMetadata": { ... },
    ...
  }
}

So you can see that updating the action flows to include the resultPath has appended their output from the AWS service onto their own keys within the input - preserving our initial state machine execution input.

This results in our final state machine execution flowing something like:

Deploy it for yourself and change the pass variable that you create an execution with and see if it flips to the other branch instead!

Remember to cdk destroy once you're done to tear down the Cloudformation stack!

Conclusion

AWS Step Functions state machine choice blocks are incredibly powerful operations that can branch your logic depending on inputs/outputs of your state machine.
AWS CDK API for provisioning these choice blocks is straightforward and easy to use.
JSON path is a critical piece of the puzzle when using state machines, handling their inputs/outputs and state transitions.
Understanding how the various paths work for inputs/outputs is very important when using state machines. AWS have good documentation on the different syntax, aspects and use cases for each of these.
They also have an interactive tool inside the AWS console itself that could be good for learning the general purpose of these - however I did find it easy to understand the concepts whenever I applied the rules to my own use case, but worth a look anyway!

How to build AWS State Machines using AWS CDK - Part I

Peter McAree — Fri, 25 Aug 2023 20:48:41 +0000

AWS Step Functions are a critical service when tying a bunch of serverless operations together. Step Functions was its own independent service developed by AWS in order to orchestrate pieces of serverless functionality together in a cost-efficient manner.

One of the most important constructs within this service are state machines. State machines are a fairly well understood concept within software engineering, and an incredibly efficient way to thinking about how your processes interact together - particularly in an event-driven architecture.

AWS Step Function state machines can be defined in a number of different ways (JSON, YAML etc.) - which is the way I've always defined them myself. However, defining these state machines as infrastructure as code to take advantage of all the things that come along with doing so, sounds like a much better approach to me!

❓ What

For those that haven't utilised AWS Step Function state machines before, they take the concept of state machines and add additional processing logic that deeply integrates with AWS serverless services in order to provide an incredibly easy interface for orchestrating complex workloads.

Think of it as a way to actually facilitate event-driven architecture using the native AWS serverless offerings, that have been enhanced to a level that can be tailored to your specific use case.

If anyone reading has ever daisy-chained lambda function invocations, state machines are something you can utilise and consider to avoid that anti-pattern

✍️ Define some CDK

Let's set up a basic state machine to perform a couple of typical actions to showcase how easy it is to define your state machine in CDK. We'll include a task state (that invokes a lambda function), a wait state, another task state (that writes to a DynamoDB table) before finishing the state machine and setting the final state.

The outcome will look something like the following:

For the purposes of this example, we'll just reference an existing lambda function and DynamoDB table that are already deployed in the AWS account you are targeting with this stack.

import * as cdk from "aws-cdk-lib";
import * as dynamodb from "aws-cdk-lib/aws-dynamodb";
import * as lambda from "aws-cdk-lib/aws-lambda";

const mockLambdaFunctionArn =
  "arn:aws:lambda:us-east-1:12345:function:my-shiny-lambda-function";
const mockDdbTableArn =
  "arn:aws:dynamodb:us-east-1:12345:table/my-shiny-dynamodb-table";

const lambdaFunction = lambda.Function.fromFunctionArn(
  this,
  "lambda-function",
  mockLambdaFunctionArn
);
const dynamodbTable = dynamodb.Table.fromTableArn(
  this,
  "dynamo-db-table",
  mockDdbTableArn
);

Now that we have our resources that we are going to trigger and reference, let's first define the lambda invocation in the state machine:

import * as stepFunctionsTasks from "aws-cdk-lib/aws-stepfunctions-tasks";

const processJob = new stepFunctionsTasks.LambdaInvoke(
  this,
  "state-machine-process-job-fn", {
    lambdaFunction: lambdaFunction,
  }
);

You may be wondering, but we're just defining a task within the state machine - where do we actually create the definition and use this task within it? Well, we'll come to that whenever we're chaining all of our actions together.

Next, we want a wait state in our definition - we can achieve this in a similar way using the CDK API:

import * as cdk from "aws-cdk-lib";
import * as stepFunctions from "aws-cdk-lib/aws-stepfunctions";

const wait10MinsTask = new stepFunctions.Wait(
  this,
  "state-machine-wait-job", {
    time: stepFunctions.WaitTime.duration(cdk.Duration.minutes(10)),
  }
);

Finally, we want to create our DynamoDB operation to round off the state machine:

import * as stepFunctionsTasks from "aws-cdk-lib/aws-stepfunctions-tasks";

const ddbWrite = new stepFunctionsTasks.DynamoPutItem(
  this,
  "ddb-write-job", {
    item: {
      uuid: stepFunctionsTasks.DynamoAttributeValue.fromString(
        crypto.randomUUID()
      ),
      timestamp: stepFunctionsTasks.DynamoAttributeValue.fromString(
        new Date().toISOString()
      ),
    },
    table: dynamodbTable,
  }
);

You'll notice that we're using the aws-stepfunctions-tasks specifically library. This has been developed in order to provide a clean interface for defining the sorts of operations that can be performed within a Task block inside the state machine (which includes operations like invoke lambda function, put item in DynamoDB table, push message to SQS queue etc.)

Now that we've defined each stage within our state machine, we can create the definition using the chainable methods before finally provisioning the construct.

import * as stepFunctions from "aws-cdk-lib/aws-stepfunctions";

const stateMachineDefinition = processJob
  .next(wait10MinsTask)
  .next(ddbWrite);

const stateMachine = new stepFunctions.StateMachine(this, "state-machine", {
  definitionBody: stepFunctions.DefinitionBody.fromChainable(
    stateMachineDefinition
  ),
  timeout: cdk.Duration.minutes(5),
  stateMachineName: "ProcessAndReportJob",
});

In the above snippet, you can see that we are chaining various stages in the state machine together using the .next() method. It consumes a reference to the next block/stage in the state machine and builds up the definition dynamically.

We then finally provision the new state machine construct and pass it this definition.

Note that the official CDK docs are slightly out of date and definition is now deprecated in favour of definitionBody. Although, this does require the additional fromChainable parsing method to be applied.

And that should be good to deploy!

🧠 Advantages

Facilitate the serverless operations in a single place, with executions of a static state machine that just processes inputs/outputs
Monitor the executions of a state machine
Fully-managed service, so don't worry about lower-level issues such as memory and CPU.

Conclusion

CDK makes it incredibly easy to define your state machines and reference all of the functionality that you require the tasks in it to perform.
Defining your state machines with CDK means that you can validate the props and references much easier than when using JSON or YAML.
CDK has a couple of supporting libraries for state machines and specifically, for the actions that task items within the state machine can support. This makes the DX super smooth and intuitive.
This is going to be the first post in a series of how we can use CDK to perform different state machine operations and define the various operations within state machines.

How to integrate Datadog with AWS ECS using AWS CDK

Peter McAree — Fri, 28 Jul 2023 10:03:35 +0000

AWS Elastic Container Service is Amazon's fully managed container orchestration service that allows you to easily deploy and scale containers. Something that you'll want to implement sooner rather than later, is some fundamental observability to keep track of your containers and ensure that everything is running as expected.

Datadog is one of the leading providers for cloud observability and the insights that it can generate out of the box after ingesting your data is second to none. They are a global partner of AWS after all!

It's been one of the providers I've used the most over the past couple of years, so have configured the agent for different compute types - one of them recently being with AWS ECS using CDK!

🏍 Sidecar pattern

When we're talking about containerised applications, we're usually building business logic as part of a service that is nicely wrapped up and portable. The beauty of containers is that they are their own little isolated sandbox and each one can execute in their own desired runtime, for example, I might have a web service running in Node.js and a backround worker running in Python. This is great for picking the correct tools for your use case and allowing them to be developed in isolation, but for certain centralised things like logging, you don't necessarily have to want to re-write the logging client in various runtimes - this is where sidecars come in.

A sidecar is a separate container that is attached to your primary container in order to provide additional functionality.

✍️ Configuring CDK

If you have used AWS ECS before, you'll be familiar with the terminology around clusters, service, task definitions and containers. Fargate is usually my go-to choice for underlying compute as well, as it means we don't have to worry about provisioning the instances under the hood!

Adding a sidecar to your primary container definition is really easy with AWS CDK, you just need to attach it to your task definition providing the container image and any other metadata required.

Note: the following code snippets assume that you have also provisioned the cluster, service, and other resources required for ECS

Firstly, you'll define your task definition:

import * as ecs from "aws-cdk-lib/aws_ecs";
import * as iam from "aws-cdk-lib/aws_iam";

const cpu = 256
const memory = 512

// Consume role from a separate ARN or provision a new one
const taskRole = iam.Role.fromRoleArn(this, "FargateTaskDefinitionRole", "<<arn>>");

const ecsFargateTaskDefinition =
    new ecs.FargateTaskDefinition(
      this,
      "FargateTaskDefinition",
      {
        cpu: cpu,
        memoryLimitMiB: memory,
        taskRole: taskRole,
      }
);

This provisions a fairly lightweight task definition with 256MB CPU and 512MB memory with a task role that we've defined somewhere else.

Next, we can add our sidecar container. Datadog have their own agent published as an image to cater for this use case, it's also pushed to the public ECR repository, so we can point our container to that image.

import * as ecs from "aws-cdk-lib/aws_ecs";

// Assume we have also attached our primary container with the same syntax as below - but pointing to our application image

const datadogContainer = ecsFargateTaskDefinition.addContainer(
  "DatadogAgentSidecarContainer",
  {
    image: ecs.ContainerImage.fromRegistry(
      "public.ecr.aws/datadog/agent:latest"
    ),
    environment: {
      ECS_FARGATE: "true",
      DD_API_KEY: process.env.DD_API_KEY,
      DD_APM_ENABLED: "true",
    },
  }
);

datadogContainer.addPortMappings({
  containerPort: 8126,
  protocol: ecs.Protocol.TCP,
});

You'll notice that as well as configuring the image to point to the public ECR repository for the image, we're also specifying some environment variables for the container - these are probably more specific depending on your set, so you can check out the docs here

Note: you can also push your Datadog API key into the likes of Secrets Manager and dynamically pull it from there, rather than just a local environment variable

Finally, we're adding the specific port mappings that the agent runs on and the protocol that it uses for communication. Make sure to check your security group configuration too, might require updated to handle this too.

And that's it! Well, pretty much. If you use custom log drivers within your other applications, you might want to specify them here as well within the container definition for the Datadog agent - but again, that's use case specific.

Run your deploy and you should have a Datadog agent sidecar container running along side your application in the same task definition!

💰 But wait, does it cost more?

I now have two containers running instead of just my single application - does this double the cost?

In terms of AWS ECS, no. Whenever you run a sidecar container, it is attached to your task definition and since your memory and CPU allocation is defined at the task definition level, it is bound by these constraints. The sidecar uses the same compute resources as your primary container.

Now in theory, if you attached a processing-intensive sidecar, you might see some side-effects and containers cycling due to running out of resources - but you've implemented Datadog for monitoring, so you'll know that before it happens, right? It's all came full circle 😄

Conclusion

Within container land, sidecars are modular, separate containers that get attached to your primary in order to provide additional functionality.
Simply attach the Datadog agent container definition to your existing (or new!) ECS task definitions.
Happy monitoring and dashboardin'.