Forem: Eric D Johnson

Modern Container Builds and WebSocket APIs Come to AWS SAM

Eric D Johnson — Tue, 05 May 2026 18:09:06 +0000

SAM CLI has always been good at taking the grunt work out of serverless deployments. You define your functions and APIs in a template, and SAM handles the CloudFormation, the packaging, the deployment. It works.

Until recently, two things were missing. BuildKit support for container image builds. And a SAM resource type for WebSocket APIs.

Both are now shipped. Neither breaks existing templates.

Let's walk through it.

BuildKit Support for Image-Based Lambda Functions
The problem
When you run sam build for an image-based Lambda function, SAM uses the docker-py Python SDK under the hood. That SDK talks to the Docker daemon directly, but it doesn't support BuildKit. At all.

That means every sam build invocation uses the legacy Docker builder. You lose parallelized build stages. You lose efficient layer caching. You lose multi-stage build optimizations. You lose cross-architecture improvements that BuildKit has shipped over the past several years.

For simple single-stage Dockerfiles, this probably doesn't matter. For anything with multiple stages, private dependencies, or cross-compilation targets, it's a real bottleneck.

What shipped
SAM CLI v1.156.0 introduced the --use-buildkit flag on sam build. When you pass this flag, SAM bypasses docker-py entirely and shells out to the Docker CLI (or Finch CLI) directly. That gives you access to everything BuildKit offers.

sam build --use-buildkit
That's it. One flag.

SAM auto-detects which container runtime you have. It defaults to Docker, falls back to Finch if Docker isn't running, and respects any admin-configured preference. Finch supports BuildKit too, so either runtime works.

What you get
BuildKit brings real improvements over the legacy builder:

Parallel stage execution. Multi-stage Dockerfiles build independent stages concurrently instead of sequentially.
Better layer caching. BuildKit tracks dependencies at the file level, not just the layer level. Change one file, rebuild one layer.
Multi-stage build optimizations. BuildKit skips stages that don't contribute to the final output. The legacy builder runs every stage regardless.
Improved cross-architecture support. Building arm64 (Graviton2) images on an x86 machine is more reliable with BuildKit's QEMU integration. Lambda currently supports Graviton2 for arm64 workloads.
Build secrets
SAM CLI v1.159.0 added support for passing BuildKit parameters, including build-time secrets. This lets you pass credentials into a build stage without baking them into a layer. Useful when your Lambda function pulls packages from a private registry during the build.

syntax=docker/dockerfile:1

FROM public.ecr.aws/lambda/python:3.12

RUN --mount=type=secret,id=pip_conf,target=/etc/pip.conf \
pip install -r requirements.txt

COPY app.py ${LAMBDA_TASK_ROOT}
CMD ["app.handler"]
You configure secrets through the Metadata section of your function resource, using DockerBuildExtraParams. SAM passes these parameters straight to docker buildx under the hood.

Resources:
MyFunction:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
Architectures: [x86_64]
Timeout: 10
Metadata:
Dockerfile: Dockerfile
DockerContext: ./src
DockerTag: latest
DockerBuildExtraParams:
- "--secret"
- "id=pip_conf,src=$HOME/.pip/pip.conf"
The secret mounts into the build stage at the path you specify in the Dockerfile, gets used during pip install, and never appears in the final image layers. Different functions can have different build secrets, since the configuration lives in each function's Metadata block.

Tradeoffs and limitations
A few things to keep in mind.

Opt-in only. This doesn't change existing build behavior. If you don't pass --use-buildkit, nothing changes. That's intentional. SAM doesn't break working builds.

Requires Docker or Finch CLI. BuildKit support works by calling the Docker or Finch CLI directly. If your CI environment only has the Docker daemon (no CLI), this won't work. Most environments have both, but check yours.

Dockerfile syntax matters. Some BuildKit features, like secret mounts, require the # syntax=docker/dockerfile:1 parser directive at the top of your Dockerfile. Without it, the build falls back to legacy parsing and you get confusing errors.

Not for ZIP-based functions. This flag only applies to image-based Lambda functions (PackageType: Image). ZIP-based functions don't use Docker at all.

The rule of thumb I like is this: if your Dockerfile is more than a few lines, or you're building for a different architecture, turn on BuildKit. The caching alone will save you time.

WebSocket API Support
The problem
Before this release, SAM had no native resource type for WebSocket APIs. You had two options: write raw CloudFormation, or don't use SAM for that part of your stack.

Here's what a minimal WebSocket API looks like in plain CloudFormation. Three routes ($connect, $disconnect, sendMessage), each backed by a Lambda function:

Resources:
WebSocketApi:
Type: AWS::ApiGatewayV2::Api
Properties:
Name: MyWebSocketApi
ProtocolType: WEBSOCKET
RouteSelectionExpression: $request.body.action

ConnectRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref WebSocketApi
RouteKey: $connect
Target: !Sub integrations/${ConnectIntegration}

ConnectIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref WebSocketApi
IntegrationType: AWS_PROXY
IntegrationUri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${ConnectFunction.Arn}/invocations

ConnectPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref ConnectFunction
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
SourceArn: !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${WebSocketApi}/*/$connect

DisconnectRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref WebSocketApi
RouteKey: $disconnect
Target: !Sub integrations/${DisconnectIntegration}

DisconnectIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref WebSocketApi
IntegrationType: AWS_PROXY
IntegrationUri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${DisconnectFunction.Arn}/invocations

DisconnectPermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref DisconnectFunction
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
SourceArn: !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${WebSocketApi}/*/$disconnect

SendMessageRoute:
Type: AWS::ApiGatewayV2::Route
Properties:
ApiId: !Ref WebSocketApi
RouteKey: sendMessage
Target: !Sub integrations/${SendMessageIntegration}

SendMessageIntegration:
Type: AWS::ApiGatewayV2::Integration
Properties:
ApiId: !Ref WebSocketApi
IntegrationType: AWS_PROXY
IntegrationUri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${SendMessageFunction.Arn}/invocations

SendMessagePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref SendMessageFunction
Action: lambda:InvokeFunction
Principal: apigateway.amazonaws.com
SourceArn: !Sub arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${WebSocketApi}/*/sendMessage

Deployment:
Type: AWS::ApiGatewayV2::Deployment
DependsOn:
- ConnectRoute
- DisconnectRoute
- SendMessageRoute
Properties:
ApiId: !Ref WebSocketApi

Stage:
Type: AWS::ApiGatewayV2::Stage
Properties:
ApiId: !Ref WebSocketApi
StageName: prod
DeploymentId: !Ref Deployment
That's twelve resources for three routes. Around 90 lines of YAML. And I left out the Lambda function definitions.

Notice the DependsOn on the Deployment resource. If you forget that, CloudFormation tries to create the deployment before the routes exist, and the stack fails. You have to manage that dependency graph yourself.

That's a lot of ceremony for "connect a WebSocket to some Lambda functions."

What shipped
The new AWS::Serverless::WebSocketApi resource type collapses all of that into this:

Resources:
MyWebSocketApi:
Type: AWS::Serverless::WebSocketApi
Properties:
RouteSelectionExpression: $request.body.action
StageName: prod
Routes:
$connect:
FunctionArn: !GetAtt ConnectFunction.Arn
$disconnect:
FunctionArn: !GetAtt DisconnectFunction.Arn
sendMessage:
FunctionArn: !GetAtt SendMessageFunction.Arn
Compare that to the CloudFormation version. Twelve resources become one. Ninety lines become twelve. The routes, integrations, Lambda permissions, deployment, stage, and resource ordering are all handled by SAM's transform.

You define the routes. SAM generates the rest.

What SAM handles automatically
For each route you declare, SAM creates:

The AWS::ApiGatewayV2::Route resource
The AWS::ApiGatewayV2::Integration wiring the route to your Lambda function
The AWS::Lambda::Permission granting API Gateway invoke access
If you add a Lambda authorizer, the authorizer permission too
SAM also creates the deployment and stage resources, with the correct dependency ordering. No DependsOn blocks to manage.

Full feature parity
This isn't a simplified subset. The new resource type supports everything API Gateway V2 WebSocket offers:

Auth: IAM authorization and Lambda authorizers, per-route or API-wide
Custom domains: Map your WebSocket API to your own domain
Route settings: Configure throttling and logging per route via RouteSettings
Models: Attach request/response models for validation
Stage variables: Pass configuration to your integration through stage variables
Globals: Share configuration across multiple WebSocket APIs using the SAM Globals section
Use cases
WebSocket APIs are the right tool when you need a persistent, bidirectional connection. Common patterns:

Chat applications. Users send and receive messages in real time.
Live dashboards. Push metric updates to connected browsers without polling.
AI/LLM streaming. Stream token-by-token responses from a model back to the client. This one is increasingly common.
IoT command channels. Send commands to devices and receive status updates on the same connection.
If your current approach is a REST API that the client polls every few seconds, a WebSocket API will give you lower latency and lower cost. Fewer requests, fewer Lambda invocations, faster updates.

Tradeoffs and limitations
No local emulation. SAM CLI doesn't support sam local start-api for WebSocket APIs. You can test individual Lambda handlers with sam local invoke, but end-to-end local WebSocket testing isn't available yet. Deploy to a dev stage for integration testing.

No sam local start-websocket. Related to the above, there's no dedicated local command for WebSocket APIs like there is for HTTP APIs with sam local start-api.

For the Agents
If you're using an AI coding agent to build SAM applications, both of these features work out of the box with agent-driven workflows. Your agent can scaffold a WebSocket API or add BuildKit to an existing image-based function without any special setup.

For Kiro, there's an official AWS SAM Power that gives your agent SAM-aware tooling. Install it and your agent gets access to sam_init, sam_build, sam_deploy, sam_logs, and sam_local_invoke as callable tools, plus opinionated project structure guidance.

Here's what a Kiro-assisted WebSocket API scaffold looks like in practice:

You: Create a WebSocket API with connect, disconnect, and sendMessage routes.
Use the SAM Power.

Kiro: [runs sam_init] → creates project structure
[updates template.yaml] → adds AWS::Serverless::WebSocketApi
[creates Lambda handlers] → connect.py, disconnect.py, send_message.py
[runs sam_build] → builds the project
[runs sam_local_invoke] → tests the connect handler locally
The SAM Power also enforces good project structure: separate Lambda handlers in infrastructure/lambda/, proper CodeUri paths, and .aws-sam in your .gitignore. It gets you from idea to deployed WebSocket API without hand-writing boilerplate.

You can install the SAM Power from the Kiro Powers marketplace or add it directly to your project's .kiro/powers/ directory.

Getting Started
Upgrade SAM CLI to get both features:

sam --version

Need v1.156.0 or later for BuildKit, latest for WebSocket APIs

Upgrade via pip

pip install --upgrade aws-sam-cli

Or via Homebrew

brew upgrade aws-sam-cli
For BuildKit, add --use-buildkit to your sam build command. No template changes needed.

For WebSocket APIs, replace your CloudFormation resources with the new AWS::Serverless::WebSocketApi type. If you're starting fresh, the SAM template above is a working starting point.

BuildKit works with sam local for local testing. WebSocket APIs currently support sam deploy for deployment and sam local invoke for testing individual handlers, but full local WebSocket emulation isn't available yet.

Two Features, Zero Breaking Changes
These two features fill gaps that have been open for a while. BuildKit support means sam build finally uses the same build engine as the rest of the container ecosystem. WebSocket API support means you can define a real-time API in SAM the same way you define a REST API. A few lines instead of a hundred.

Neither feature changes existing behavior. Both are additive. Upgrade, try them, and keep building.

Cold Starts Are Dead

Eric D Johnson — Wed, 29 Apr 2026 03:09:17 +0000

It never fails. Every time I talk about serverless, someone pushes back with the cold start argument. I still see it in forums, in blog comments, in architecture review meetings. "Sure, but what about cold starts?"

I get it. Five or six years ago, that was a legitimate concern.

But it's 2026. The data tells a different story. And if you're still making decisions based on the cold start argument, you're arguing against a version of Lambda that hasn't existed in years.

How Long Are Lambda Cold Starts in 2026?

Let's start with what cold starts actually look like today. These numbers come from production workloads observed in the wild, not synthetic hello-world tests. Your mileage will vary by package size, initialization code, and memory configuration, but the ranges are representative of what teams are seeing in 2026.

Runtime	P50	P99	Notes
Python 3.13	200-400ms	800ms-1.2s	Fastest scripting runtime
Node.js 22	200-350ms	600ms-1s	Solid general choice
Go	50-100ms	150-250ms	Near-zero
Rust	50-80ms	100-200ms	Fastest overall
Java 21 (no SnapStart)	2-5s	6-10s	Still slow without SnapStart
Java 21 + SnapStart	90-140ms	200-400ms	Dramatically better

Running on arm64 (Graviton)? Knock another 15-40% off those numbers across the board.

Rust cold starts have been measured as low as 16ms on arm64. Sixteen milliseconds, which is less a cold start problem and more a rounding error.

The scripting runtimes, Python and Node, land in the 200-400ms range at P50. For context, that's less than the time it takes your browser to render a page after receiving the HTML.

Does VPC Still Cause Lambda Cold Starts?

This one still comes up in conversations, and it frustrates me because it was solved in 2019.

The old problem: when a Lambda function needed VPC connectivity, AWS had to create an Elastic Network Interface (ENI) on the fly. That meant 10-15 seconds of additional cold start latency on top of everything else. VPC-connected Lambda was genuinely painful.

AWS fixed this when Lambda migrated to Firecracker microVMs in 2019, dropping cold start overhead from over ten seconds to under a second. Werner Vogels recently wrote about the invisible engineering behind Lambda's network. The team used eBPF to rewrite Geneve tunnel headers, taking tunnel latency from 150 milliseconds to 200 microseconds. The VPC cold start penalty now approaches zero for most workloads.

That was seven years ago. If someone tells you Lambda cold starts are bad because of VPC, they're working from outdated information.

How Much Does SnapStart Reduce Cold Starts?

Java was the poster child for the cold start argument. And honestly, it earned that reputation. A Spring Boot app on Lambda could take 3-10 seconds to cold start. That's painful no matter how you frame it.

SnapStart changed the math. Here's the timeline:

November 2022: SnapStart GA for Java at re:Invent
November 2024: Python 3.12+ and .NET 8+ GA
2025: Expanded to additional regions, arm64 support

How it works: Lambda takes a snapshot of the initialized Firecracker microVM after your INIT code runs. That snapshot gets cached across three tiers: L1 on the worker, L2 in the placement group, S3 at the region level. On cold start, Lambda restores from the snapshot instead of re-running your initialization code.

The benchmarks are real:

Python (Flask, LangChain, Pandas): several seconds → sub-second
Java Spring Boot: 5.8s → 180ms (97% reduction)
.NET: 58-94% cold start reduction

SnapStart has continued to evolve. Python and .NET support went GA in late 2024, with additional regions and arm64 support following in 2025.

Did the Lambda INIT Billing Change Increase Costs?

In August 2025, AWS standardized billing for the Lambda INIT phase. Functions packaged as ZIP files with managed runtimes now get billed for the INIT phase, which was previously free. This triggered some alarming blog posts.

The headline claim: a "22x cost increase." Let's look at the math.

That 22x number requires a perfect storm of worst-case assumptions:

100% cold start rate (every invocation is a cold start)
2-second Java INIT duration
512MB memory configuration

Here's the problem with that scenario: AWS's own production analysis shows cold starts occur in less than 1% of invocations. Not 100%. Less than 1%.

AWS's own assessment: "most users will see minimal impact on their overall Lambda bill from this change, as the INIT phase typically occurs for a very small fraction of function invocations." The actual impact depends on your cold start ratio and INIT duration relative to handler duration. Use the CloudWatch query below to check your own numbers.

Don't take my word for it. AWS published a CloudWatch Logs Insights query so you can calculate your exact impact:

filter @type = "REPORT"
| stats
    sum((@memorySize/1000000/1024) * (@billedDuration/1000)) as BilledGBs,
    sum((@memorySize/1000000/1024) * ((@duration + @initDuration - @billedDuration)/1000)) as UnbilledInitGBs,
    UnbilledInitGBs / (UnbilledInitGBs + BilledGBs) as UnbilledInitRatio

Run that against your own functions. If your UnbilledInitRatio is anywhere near the number that would produce a 22x increase, you have a bigger problem than billing changes. You have an architecture problem.

With SnapStart enabled, INIT durations drop to sub-second, which shrinks that ratio even further.

I Decided to See for Myself

I wasn't satisfied citing someone else's numbers for a post called "Cold Starts Are Dead." So I built a benchmarker. Thirteen Lambda functions across six runtimes, both arm64 and x86_64, 50 cold start invocations each, 512MB memory, us-west-2. Minimal hello-world handlers, no frameworks, no dependencies beyond the runtime SDK. This measures the platform floor, not application init time.

Runtime	Arch	P50 (ms)	P99 (ms)	Blog Claim	Verdict
Rust	arm64	14.1	31.9	50-80ms	⚡ Faster
Rust	x86_64	17.0	29.3	50-80ms	⚡ Faster
Go	arm64	45.0	61.2	50-100ms	⚡ Faster
Go	x86_64	59.8	94.9	50-100ms	✅ Verified
Python 3.13	arm64	88.3	147.1	200-400ms	⚡ Faster
Python 3.13	x86_64	106.2	142.3	200-400ms	⚡ Faster
Node.js 22	arm64	121.5	168.5	200-350ms	⚡ Faster
Node.js 22	x86_64	155.0	231.3	200-350ms	⚡ Faster
Java 21	arm64	365.3	539.2	2-5s	⚡ Faster
Java 21	x86_64	443.8	573.5	2-5s	⚡ Faster

Every runtime came in at or below the production ranges I cited earlier. That's expected. Those production numbers include application dependencies, framework initialization, and SDK client setup. My minimal handlers isolate just the platform overhead. Think of these as the floor: your cold starts will be at least this fast, plus whatever your initialization code adds.

The arm64 advantage verified cleanly too:

Runtime	arm64 P50	x86_64 P50	Improvement
Rust	14.1ms	17.0ms	17% faster
Go	45.0ms	59.8ms	25% faster
Python	88.3ms	106.2ms	17% faster
Node.js	121.5ms	155.0ms	22% faster
Java	365.3ms	443.8ms	18% faster

17-25% faster across every runtime. If you're still deploying on x86_64 in 2026, you're leaving performance on the table for no reason.

I also tested VPC vs non-VPC with the Python function. The VPC-connected function was 1.4ms faster at P50, within noise.

One honest caveat: SnapStart on my minimal Java handler showed a ~670ms restore duration. That's because there's almost nothing to snapshot. The restore mechanism's own overhead dominates. For a Spring Boot app where SnapStart eliminates 3-5 seconds of framework init, you'd see the dramatic improvement the benchmarks above describe. SnapStart's value scales with how much init work your app does.

The full benchmarker is open source on GitHub. You can run it yourself:

git clone https://github.com/singledigit/lambda-benchmark.git
cd lambda-benchmark
sam build && sam deploy --guided
cd orchestrator
python benchmark.py --stack-name <stack> --iterations 50
python generate_report.py

When Do Lambda Cold Starts Still Matter?

Here's where cold starts still matter.

But first, is 200ms even slow? For a website, a chat app, an API powering a mobile experience, an AI agent? You won't notice it. A typical page load involves DNS resolution, TLS handshake, and content rendering that add up to far more than 200ms. If you're building AI agents, the LLM call alone takes 2-10 seconds, so 200ms of cold start is a rounding error on that. It's the first request after a quiet period, and it happens in less time than a blink.

The cases where cold starts actually matter are narrow:

Sub-10ms latency SLAs. If you're building for high-frequency trading or real-time bidding, 200ms of cold start latency is unacceptable. At that point, you're probably looking at containers on ECS or EKS where you control the warm pool directly. Provisioned Concurrency is an option too, but it has its own cost tradeoffs.

Very spiky, unpredictable traffic with strict P99 requirements. If your traffic pattern goes from zero to thousands of concurrent invocations with no ramp-up, and you have hard P99 latency SLAs, cold starts will hit that tail. You have options: containers with pre-scaled task counts, Provisioned Concurrency, or accepting slightly higher P99s during the initial burst. The right answer depends on your cost tolerance and how strict "strict" actually is.

Java without SnapStart. Just... use SnapStart. If you're running Java on Lambda without SnapStart enabled, you're choosing to have a cold start problem. The fix is a configuration change.

The honest framing: the question isn't "do cold starts exist?" They do. The question is "do they matter for your workload?" For the vast majority of workloads, the answer is no.

Conclusion

VPC cold starts were fixed in 2019, and SnapStart took Java from 5 seconds to 180 milliseconds. Python and Node sit under 200ms on arm64, and AWS's own data says cold starts happen in less than 1% of invocations. I built a benchmarker and tested it myself. The production claims are conservative, and the numbers are better than advertised.

The cold start argument had its day. That day was 2018. If you're still leading with it, I get it, it used to be real. But the data has moved on, and it's time we did too.

For the Agents

If you're using an AI coding assistant or agent, I've put together a companion skill file for this post. It includes the 2026 cold start benchmarks, SnapStart configurations, and the CloudWatch query for INIT billing analysis. Everything your agent needs to give you accurate cold start guidance instead of outdated advice.

Is this code deterministic?

Eric D Johnson — Fri, 16 Jan 2026 18:20:30 +0000

I recently posted a small code snippet in a LinkedIn poll and asked what sounded like a simple question:

Is this code deterministic?

Those are usually the dangerous questions.

I asked on purpose. I’ve been spending time talking with folks much smarter than me, reading docs, and honestly leaning on code assistants to sanity-check my thinking as I go. Durable execution has a way of surfacing edge cases you don’t normally think about, and I wanted to learn in public—right alongside everyone else.

The discussion that followed (in the original post) was excellent. It also showed how easy it is to mix together concepts like determinism, replay, retries, and idempotency. This post is my attempt to slow things down and separate those ideas, using the original example and AWS’s guidance on deterministic code in AWS Lambda durable functions.

Here’s the code that started it all:

import { withDurableExecution, DurableContext } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const orders = event.orders.sort((a, b) => a.priority - b.priority);

    const results = [];
    for (const order of orders) {
      const result = await context.step(`process-${order.id}`, async () => {
        return processOrder(order);
      });
      results.push(result);
    }

    return { processed: results.length, timestamp: Date.now() };
  }
);

Most people voted “No, non-deterministic.”

That’s the correct answer—but not always for the reasons people first reach for.

Let’s walk through it.

Problem 1: Equal-priority ordering is under-specified

What’s happening

event.orders.sort((a, b) => a.priority - b.priority);

Modern JavaScript engines (ES2019+) guarantee that Array.prototype.sort() is stable. If two orders have the same priority, their relative order is preserved.

So no, JavaScript isn’t secretly reordering your data.

Why it still matters (and why this one is subtle)

I’ll be honest: this one felt nit-picky to me at first. If the input is the same and the sort is stable, it feels like everything should be fine.

The important realization is this: a stable sort preserves whatever order the input already had—but it doesn’t explain why that order exists.

In this code, the implicit rule becomes:

“If priorities are equal, keep whatever order the input arrived in.”

If that order is intentional and guaranteed, great. Nothing wrong here.

But if it’s incidental—maybe merged upstream, aggregated from multiple sources, or simply not meant to be meaningful—then the workflow’s step ordering now depends on an accident of the input.

Nothing is broken. But you may have just encoded behavior you didn’t mean to encode.

One more thing worth saying out loud: if the order in which steps are created doesn’t matter, you may not need to sort at all. Sorting only makes sense if you’re enforcing a real business rule like “highest priority goes first.”

Why not just wrap the ordering in a step?

This is a very common reaction—and a reasonable one.

Yes, you could wrap the sort in a step and checkpoint it. That would make the ordering fully durable and replay-stable.

But steps are not free.

They add latency. They cost money. They count toward operation limits. And they exist primarily to protect work that is slow, expensive, or has side effects.

Pure, fast, in-memory logic like sorting is already replayable. Re-running a sort of 10 items during replay is usually far cheaper than checkpointing it. Even with larger lists, the trade-off depends on size, cost, and intent.

The rule of thumb I like is this:

If the logic is pure, fast, and deterministic, don’t rush to wrap it.

If you can’t make it deterministic, or replaying it is expensive, that’s when a step makes sense.

Fix

If ordering matters, make it explicit and deterministic, without mutating the input:

const orders = [...event.orders].sort(
  (a, b) => a.priority - b.priority || a.id.localeCompare(b.id)
);

Problem 2: `Date.now()` is non-deterministic

What’s happening

timestamp: Date.now()

This value is computed at runtime, so every execution produces a different number.

Why it matters

In this handler, the timestamp is just part of the returned response. It doesn’t affect control flow or step scheduling, so it’s harmless today.

But time-based APIs are explicitly called out in the durable execution docs as a common source of non-determinism. If this value later gets stored as workflow state, passed into a step, or used in a conditional, replay behavior can change in ways that are very hard to reason about.

This is less “this is wrong” and more “this is easy to trip over later.”

Fix

If the timestamp actually matters, capture it once inside a step so it replays consistently:

const timestamp = await context.step("timestamp", async () => Date.now());

Problem 3: Side effects hidden inside a single step

Ben Kehoe correctly points out a subtle but important issue in this code.

What’s happening

await context.step(`process-${order.id}`, async () => {
  return processOrder(order);
});

A durable step can fail and be retried. Once a step completes, it won’t be re-run on replay—but retries can re-execute the step body.

If processOrder performs multiple side effects, a failure partway through can cause those side effects to run again.

Why it matters

This is not a determinism problem.

It’s not a replay problem either.

This is a retry safety problem.

If a step body can’t safely run more than once, retries can produce duplicate effects unless everything inside the step is idempotent.

Fix

Be intentional about retry boundaries and align steps with retry-safe work.

Problematic version:

await context.step(`process-${order.id}`, async () => {
  await chargeCard(order);
  await writeAuditRecord(order);
  await sendConfirmation(order);
});

If this fails after chargeCard, a retry may re-run everything.

Safer version:

await context.step(`charge-${order.id}`, async () => {
  return chargeCard(order, { idempotencyKey: order.id });
});

await context.step(`audit-${order.id}`, async () => {
  return writeAuditRecord(order);
});

await context.step(`notify-${order.id}`, async () => {
  return sendConfirmation(order, { idempotencyKey: order.id });
});

This doesn’t magically make things idempotent. It just limits the blast radius when retries happen.

Step semantics and retry intent

AWS Lambda durable functions also give you control over how steps retry.

By default, steps use AtLeastOncePerRetry semantics. If a step fails or the Lambda is interrupted, the runtime may re-execute the step body. In this mode, the retry count acts as a lower bound on executions.

If you have a step that must never run more than once, you can use StepSemantics.AtMostOncePerRetry with zero retries. In that case, a failure surfaces as an error instead of re-running the step.

Put simply:

AtLeastOncePerRetry → max attempts is a lower bound
AtMostOncePerRetry → max attempts is an upper bound

Neither is “better.” They just encode different assumptions.

Putting it all together

Once you make ordering explicit, keep non-deterministic values under control, and think carefully about retry boundaries, the handler becomes much easier to reason about.

Here are two durable-safe ways to structure it, depending on how independent your work items are and how much concurrency you want.

Durable-safe handler (step by step)

import { withDurableExecution, DurableContext } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const orders = [...event.orders].sort(
      (a, b) => a.priority - b.priority || a.id.localeCompare(b.id)
    );

    for (const order of orders) {
      await context.step(`validate-${order.id}`, async () => {
        if (!order.id) throw new Error("Missing order id");
      });

      await context.step(`charge-${order.id}`, async () => {
        return chargeCard(order, { idempotencyKey: order.id });
      });

      await context.step(`notify-${order.id}`, async () => {
        return sendConfirmation(order, { idempotencyKey: order.id });
      });
    }

    return { processed: orders.length };
  }
);

Durable-safe handler using `context.map()`

context.map() changes the shape of the problem a bit. Each item becomes its own durable unit of work.

That matters because:

Failures are isolated to a single item
Completed items don’t get re-run because something else failed
Concurrency becomes a first-class knob (maxConcurrency)

The trade-offs are real too:

Large lists can emit a lot of steps quickly
Strict sequencing is harder to express

import { withDurableExecution, DurableContext } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const orders = [...event.orders].sort(
      (a, b) => a.priority - b.priority || a.id.localeCompare(b.id)
    );

    const mapResult = await context.map(
      "process-orders",
      orders,
      async (ctx: DurableContext, order: any) => {
        await ctx.step(`validate-${order.id}`, async () => {
          if (!order.id) throw new Error("Missing order id");
        });

        await ctx.step(`charge-${order.id}`, async () => {
          return chargeCard(order, { idempotencyKey: order.id });
        });

        await ctx.step(`notify-${order.id}`, async () => {
          return sendConfirmation(order, { idempotencyKey: order.id });
        });

        return { orderId: order.id, status: "ok" };
      },
      { maxConcurrency: 5 }
    );

    const results = mapResult.getResults();
    return { processed: results.length, results };
  }
);

Final takeaway

Durable execution encourages you to slow down just a bit and be explicit about ordering, retries, idempotency, and where work can safely be repeated.

That’s exactly why this question was worth asking—and why the conversation around it was worth having.

AWS Lambda Durable Functions: Build Workflows That Last

Eric D Johnson — Wed, 03 Dec 2025 21:33:02 +0000

Long-running workflows without managing infrastructure

Your Lambda function needs to wait for a human approval. Or retry a failed API call with exponential backoff. Or orchestrate multiple steps that span hours. How do you build that without managing servers or databases?

AWS Lambda Durable Functions solve this. Write your workflow in your programming language—Node.js, TypeScript, Python, with more coming—using straightforward async code. Lambda handles the rest: checkpointing state, resuming after waits, retrying issues, and scaling automatically. Workflows can run for up to a year, and you only pay for actual execution time—not while waiting.

What Are Durable Functions?

Durable functions are Lambda functions that can pause and resume. When your function waits for a callback or sleeps for an hour, Lambda checkpoints its state and stops execution. When it's time to continue, Lambda resumes exactly where it left off—with all variables and context intact.

This isn't a new compute model. It's regular Lambda with automatic state management. You write normal async/await code. Lambda makes it durable.

A Simple Example

Here's a workflow that creates an order, waits 5 minutes, then sends a notification:

import { DurableContext, withDurableExecution } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const order = await context.step('create-order', async () => {
      return createOrder(event.items);
    });

    await context.wait({ seconds: 300 });

    await context.step('send-notification', async () => {
      return sendEmail(order.customerId, order.id);
    });

    return { orderId: order.id, status: 'completed' };
  }
);

That's it. No state machines to configure, no databases to manage, no polling loops. The function pauses during the wait, costs nothing while idle, and resumes automatically after 5 minutes.

Key Capabilities

Long execution times - Functions can run for up to 1 year. Individual invocations are still limited to 15 minutes, but the workflow continues across multiple invocations.

Automatic checkpointing - Lambda saves your function's state at each step. If something fails, the function resumes from the last checkpoint—not from the beginning.

Built-in retries - Configure retry strategies with exponential backoff. Lambda handles the retry logic and timing automatically.

Wait for callbacks - Pause execution until an external event arrives. Perfect for human approvals, webhook responses, or async API results.

Parallel execution - Run multiple operations concurrently and wait for all to complete. Lambda manages the coordination.

Nested workflows - Invoke other durable functions and compose complex workflows from simple building blocks.

How It Works: The Replay Model

Durable functions use a replay-based execution model. When your function resumes, Lambda replays it from the start—but instead of re-executing operations, it uses checkpointed results.

Here's what happens:

First invocation - Your function runs, executing each step and checkpointing results
Wait or callback - Function pauses, Lambda saves state and stops execution
Resume - Lambda invokes your function again, replaying from the start
Replay - Operations return checkpointed results instantly instead of re-executing
Continue - Function continues past the wait with all context intact

This model ensures your function always sees consistent state, even across issues and restarts. Operations are deterministic—they execute once and replay with the same result.

Learn more: Understanding the Replay Model explains how replay works, why operations must be deterministic, and how to handle non-deterministic code safely.

Common Use Cases

Approval workflows - Wait for human approval before proceeding. The function pauses until someone clicks approve or reject.

Saga patterns - Coordinate distributed transactions with compensating actions. If a step fails, automatically roll back previous steps.

Scheduled tasks - Wait for specific times or intervals. Process data at midnight, send reminders after 24 hours, or retry every 5 minutes.

API orchestration - Call multiple APIs with retries and error handling. Coordinate responses and handle partial issues gracefully.

Data processing pipelines - Process large datasets in stages with checkpoints. Resume from the last successful stage if something fails.

Event-driven workflows - React to external events like webhooks, IoT signals, or user actions. Wait for events and continue processing when they arrive.

Testing Your Workflows

Testing long-running workflows doesn't mean waiting hours. The Durable Execution SDK includes a testing library that runs your functions locally in milliseconds:

import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';

const runner = new LocalDurableTestRunner({
  handlerFunction: handler,
});

const execution = await runner.run();

expect(execution.getStatus()).toBe('SUCCEEDED');
expect(execution.getResult()).toEqual({ orderId: '123', status: 'completed' });

The test runner simulates checkpoints, skips time-based waits, and lets you inspect every operation. You can test callbacks, retries, and issues without deploying to AWS.

Learn more: Testing Durable Functions covers local testing, cloud integration tests, debugging techniques, and best practices.

Deploying with AWS SAM

Deploy durable functions using AWS SAM with a few key configurations:

Resources:
  OrderProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/order-processor
      Handler: index.handler
      Runtime: nodejs22.x
      DurableConfig:
        ExecutionTimeout: 900
        RetentionPeriodInDays: 7
    Metadata:
      BuildMethod: esbuild
      BuildProperties:
        EntryPoints:
          - index.ts

The DurableConfig property enables durable execution and sets the workflow timeout. SAM automatically handles IAM permissions for checkpointing and state management.

Learn more: Deploying Durable Functions with SAM covers template configuration, permissions, build settings, and deployment best practices.

When to Use Durable Functions

Your workflow spans multiple steps with waits or callbacks
You need automatic retries with exponential backoff
You want to coordinate multiple async operations
Your process requires human approval or external events
You need to handle long-running tasks without managing state
You prefer writing workflows as code rather than configuration

Getting Started

Install the SDK: npm install @aws/durable-execution-sdk-js
Write your function: Wrap your handler with withDurableExecution()
Use durable operations: context.step(), context.wait(), context.waitForCallback()
Test locally: Use LocalDurableTestRunner for fast iteration
Deploy with SAM: Add DurableConfig to your template
Monitor execution: Use Amazon CloudWatch and AWS X-Ray for observability

Learn More

Understanding the Replay Model - Deep dive into how durable functions work under the hood
Testing Durable Functions - Comprehensive guide to testing workflows locally and in the cloud
Deploying with AWS SAM - Complete deployment guide with templates and best practices

Summary

AWS Lambda Durable Functions let you build long-running workflows without managing infrastructure. Write straightforward async code, and Lambda handles state management, retries, and resumption. Your functions can wait for callbacks, retry issues, and run for up to a year—all while paying only for execution time.

Start with simple workflows, test locally for fast iteration, and deploy with confidence knowing Lambda manages the complexity of distributed state.

Developing AWS Lambda Durable Functions with AWS SAM

Eric D Johnson — Wed, 03 Dec 2025 21:25:35 +0000

How to configure, build, and deploy long-running workflows using SAM templates

You've written a durable function that orchestrates a multi-step workflow. Now you may want to deploy it. AWS Serverless Application Model (AWS SAM) makes this straightforward with specific configurations in your template. Let's walk through how to set up durable functions effectively.

Prerequisites

You'll need AWS SAM CLI version 1.150.1 or greater. Check your version:

sam --version

If you need to upgrade, follow the AWS SAM installation guide.

The Basics: Enabling Durable Functions

To make an AWS Lambda function durable, add the DurableConfig property to your AWS SAM template:

Resources:
  OrderProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/order-processor
      Handler: index.handler
      Runtime: nodejs22.x
      Architectures:
        - arm64
      Timeout: 900                # Function timeout: 15 minutes
      DurableConfig:
        ExecutionTimeout: 3600    # Execution timeout: 1 hour
        RetentionPeriodInDays: 7  # Keep execution history for 7 days

Understanding the two timeout settings is crucial:

Function Timeout (Timeout) controls how long each individual Lambda invocation can run. This is still capped at 15 minutes (900 seconds), just like regular Lambda functions. Each time your durable function checkpoints and resumes, it's a new invocation with its own 15-minute window.

Execution Timeout (ExecutionTimeout) controls how long the entire workflow can run across all invocations. This can be up to 1 year. Your workflow can pause, wait for callbacks, and resume many times, as long as the total elapsed time doesn't exceed this limit.

For long-running workflows where the execution timeout exceeds the function timeout, you must invoke the function asynchronously. Synchronous invocations will fail with a validation error if the execution timeout is longer than the function timeout.

Installing the SDK

Add the durable execution SDK to your function's dependency manifest. SAM will automatically install it during the build process.

TypeScript/JavaScript - Add to package.json:

{
  "dependencies": {
    "@aws/durable-execution-sdk-js": "^1.0.0"
  }
}

Python - Add to requirements.txt:

aws-durable-execution-sdk-python

For client functions that send callbacks or query execution state, add the AWS SDK:

TypeScript/JavaScript:

{
  "dependencies": {
    "@aws-sdk/client-lambda": "^3.0.0"
  }
}

Python:

boto3

Build Configuration

Configure SAM to build your TypeScript functions with esbuild:

Resources:
  OrderProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/order-processor
      Handler: index.handler
      Runtime: nodejs22.x
      Architectures:
        - arm64
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      Metadata:
        BuildMethod: esbuild
        BuildProperties:
          EntryPoints:
            - index.ts

SAM will automatically compile your TypeScript code and bundle dependencies into a deployment package.

IAM Permissions

AWS SAM automatically grants the necessary permissions for durable functions to checkpoint and manage their execution state. You don't need to explicitly configure these permissions.

However, you may want to configure permissions for client functions that interact with durable executions. For functions that send callbacks:

BaristaCallbackFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: src/barista-callback
    Handler: index.handler
    Policies:
      - Statement:
          - Effect: Allow
            Action:
              - lambda:SendDurableExecutionCallbackSuccess
              - lambda:SendDurableExecutionCallbackFailure
              - lambda:SendDurableExecutionCallbackHeartbeat
            Resource: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${CoffeeOrderFunction}'

For functions that monitor execution state or stop executions:

MonitoringFunction:
  Type: AWS::Serverless::Function
  Properties:
    CodeUri: src/monitoring
    Handler: index.handler
    Policies:
      - Statement:
          - Effect: Allow
            Action:
              - lambda:GetDurableExecution
              - lambda:GetDurableExecutionHistory
              - lambda:ListDurableExecutionsByFunction
              - lambda:SendDurableExecutionCallbackSuccess
              - lambda:SendDurableExecutionCallbackHeartbeat
              - lambda:SendDurableExecutionCallbackFailure
              - lambda:StopDurableExecution
            Resource: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${CoffeeOrderFunction}'

These permissions allow client functions to interact with your durable workflows by sending callbacks, monitoring execution state, or stopping running executions.

Using Globals for Common Configuration

AWS SAM's Globals section reduces repetition when you have multiple functions with shared settings:

Globals:
  Function:
    Runtime: nodejs22.x
    Architectures:
      - arm64
    Timeout: 30
    MemorySize: 512
    Tracing: Active

Resources:
  OrderProcessorFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/order-processor
      Handler: index.handler
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7

  CallbackHandlerFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/callback-handler
      Handler: index.handler
      # No DurableConfig - this is a regular Lambda function

You can set common runtime, architecture, and memory settings at the global level. For DurableConfig, configure it at the function level to make it explicit which functions are durable. Functions without DurableConfig are regular Lambda functions.

A Complete Example

Here's a full AWS SAM template for a coffee ordering system with durable workflows:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Coffee Ordering System with Durable Functions

Globals:
  Function:
    Runtime: nodejs22.x
    Architectures:
      - arm64
    Timeout: 30
    MemorySize: 512
    Tracing: Active

Resources:
  # ========== AMAZON DYNAMODB ==========

  OrdersTable:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: CoffeeOrders
      BillingMode: PAY_PER_REQUEST
      AttributeDefinitions:
        - AttributeName: orderId
          AttributeType: S
      KeySchema:
        - AttributeName: orderId
          KeyType: HASH

  # ========== FUNCTIONS ==========

  CoffeeOrderFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/coffee-order
      Handler: index.handler
      AutoPublishAlias: Live
      DurableConfig:
        ExecutionTimeout: 3600
        RetentionPeriodInDays: 7
      Environment:
        Variables:
          ORDERS_TABLE: !Ref OrdersTable
      Policies:
        - DynamoDBCrudPolicy:
            TableName: !Ref OrdersTable
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /orders
            Method: post
      Metadata:
        BuildMethod: esbuild
        BuildProperties:
          EntryPoints:
            - index.ts

  BaristaCallbackFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/barista-callback
      Handler: index.handler
      Policies:
        - Statement:
            - Effect: Allow
              Action:
                - lambda:SendDurableExecutionCallbackSuccess
                - lambda:SendDurableExecutionCallbackFailure
              Resource: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${CoffeeOrderFunction}'
      Events:
        ApiEvent:
          Type: Api
          Properties:
            Path: /barista/accept/{orderId}
            Method: post
      Metadata:
        BuildMethod: esbuild
        BuildProperties:
          EntryPoints:
            - index.ts

Outputs:
  ApiUrl:
    Description: API Gateway endpoint URL
    Value: !Sub 'https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod'

  CoffeeOrderFunctionArn:
    Description: Coffee Order Function ARN
    Value: !GetAtt CoffeeOrderFunction.Arn

Building and Deploying

Build your application with AWS SAM:

sam build

This builds your functions and layers. AWS SAM automatically runs the Makefiles for your layers and uses esbuild for your TypeScript functions.

Deploy to AWS:

sam deploy --guided

The --guided flag walks you through configuration options. After the first deployment, AWS SAM saves your settings in samconfig.toml, so subsequent deploys are just:

sam deploy

Local Testing

Test your durable function locally before deploying:

sam local invoke CoffeeOrderFunction --event events/order.json

The event file contains your test payload:

{
  "orderId": "test-123",
  "attendeeId": "user-456",
  "orderDetails": {
    "drinkType": "Latte",
    "size": "Grande"
  }
}

Accessing Cloud Resources Locally

When testing locally, you often need to access deployed AWS resources like Amazon DynamoDB tables or Amazon EventBridge buses. Use the --env-vars flag with a JSON file to provide environment variables and credentials:

sam local invoke CoffeeOrderFunction \
  --event events/order.json \
  --env-vars locals.json

Create a locals.json file with environment variables for each function:

{
  "CoffeeOrderFunction": {
    "AWS_REGION": "us-east-1",
    "ORDERS_TABLE_NAME": "CoffeeOrders",
    "EVENT_BUS_NAME": "CoffeeOrderingEventBus"
  },
  "BaristaCallbackFunction": {
    "AWS_REGION": "us-east-1",
    "ORDERS_TABLE_NAME": "CoffeeOrders"
  }
}

This allows your local function to interact with deployed resources in AWS, assuming your AWS credentials have the necessary permissions. Your function code can access these variables through process.env:

const tableName = process.env.ORDERS_TABLE_NAME;
const region = process.env.AWS_REGION;

Tracking Execution State

For durable functions, you can track execution state:

# Get execution details
sam local execution get $EXECUTION_ARN

# View execution history
sam local execution history $EXECUTION_ARN

# Stop a running execution
sam local execution stop $EXECUTION_ARN

Test callbacks locally:

# Send success callback
sam local callback succeed $CALLBACK_ID --result '{"status": "accepted"}'

# Send failure callback
sam local callback fail $CALLBACK_ID --error '{"message": "Rejected"}'

# Send heartbeat
sam local callback heartbeat $CALLBACK_ID

Remote Testing

After deploying, test against your live functions:

# Invoke the function (automatically uses $LATEST qualifier)
sam remote invoke CoffeeOrderFunction --event events/order.json

# Invoke a specific qualifier/alias
sam remote invoke CoffeeOrderFunction --event events/order.json --parameter Qualifier=prod

# Get execution details
sam remote execution get $EXECUTION_ARN

# View execution history
sam remote execution history $EXECUTION_ARN

By default, sam remote invoke uses the $LATEST qualifier. You can override this with --parameter Qualifier=<your-qualifier> to test against a specific version or alias.

The execution history shows every step, checkpoint, and state transition. It's invaluable for debugging workflows in production.

Configuration Best Practices

ExecutionTimeout can be up to 1 year (31,536,000 seconds) and controls how long your entire workflow can run across all invocations. For a coffee order that waits up to 5 minutes for barista acceptance, consider setting it to 600 seconds (10 minutes) to allow some buffer. For document processing that might take hours or days, you can set it much higher - up to the 1-year maximum.

Function Timeout is still capped at 15 minutes (900 seconds) and controls how long each individual invocation can run. Set this based on how long your function needs between checkpoints. If your steps typically complete in seconds, starting with 30 seconds is reasonable. If you have longer-running operations, consider increasing it up to the 15-minute maximum.

Async vs Sync Invocation: If your execution timeout exceeds your function timeout, you must use asynchronous invocation. Synchronous invocations will fail validation when the execution timeout is longer than the function timeout. Configure your event sources (API Gateway, EventBridge, etc.) to invoke asynchronously for long-running workflows.

RetentionPeriodInDays determines how long execution history is kept. Consider using 7-30 days for development environments where you're actively debugging. For production environments where you might need to investigate issues weeks later, consider using 90+ days.

Memory affects both performance and cost. Starting with 512 MB and adjusting based on your function's needs is a common approach. Durable functions checkpoint frequently, so memory usage is typically lower than equivalent non-durable functions.

Monitoring and Observability

Enable AWS X-Ray tracing to see how your workflow executes:

Globals:
  Function:
    Tracing: Active

This traces each invocation and shows you the complete execution path across multiple invocations.

Add Amazon CloudWatch Logs Insights queries to analyze execution patterns:

fields @timestamp, @message
| filter @message like /CHECKPOINT/
| sort @timestamp desc
| limit 100

Create CloudWatch alarms for execution failures:

ExecutionFailureAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: CoffeeOrderExecutionFailures
    MetricName: Errors
    Namespace: AWS/Lambda
    Statistic: Sum
    Period: 300
    EvaluationPeriods: 1
    Threshold: 5
    ComparisonOperator: GreaterThanThreshold
    Dimensions:
      - Name: FunctionName
        Value: !Ref CoffeeOrderFunction

Common Patterns

Event-driven workflows triggered by API Gateway, EventBridge, or Amazon Simple Queue Service (Amazon SQS) - useful for order processing, approval workflows, and asynchronous task execution:

CoffeeOrderFunction:
  Type: AWS::Serverless::Function
  Properties:
    # ... other properties
    Events:
      ApiTrigger:
        Type: Api
        Properties:
          Path: /orders
          Method: post
      EventBridgeTrigger:
        Type: EventBridgeRule
        Properties:
          Pattern:
            source:
              - coffee.orders
            detail-type:
              - OrderPlaced

For long-running workflows, configure API Gateway to invoke asynchronously:

CoffeeOrderFunction:
  Type: AWS::Serverless::Function
  Properties:
    # ... other properties
    Timeout: 900  # 15 minutes
    DurableConfig:
      ExecutionTimeout: 86400  # 24 hours
    Events:
      ApiTrigger:
        Type: Api
        Properties:
          Path: /orders
          Method: post
          RequestParameters:
            - method.request.header.X-Amz-Invocation-Type:
                Required: false
                Caching: false

Scheduled workflows that run on a timer - useful for daily reports, periodic data processing, or scheduled maintenance tasks:

DailyReportFunction:
  Type: AWS::Serverless::Function
  Properties:
    # ... other properties
    Events:
      DailySchedule:
        Type: Schedule
        Properties:
          Schedule: cron(0 9 * * ? *)  # 9 AM daily

Fan-out workflows that process items in parallel - useful for batch processing, data transformation, or concurrent API calls:

BatchProcessorFunction:
  Type: AWS::Serverless::Function
  Properties:
    # ... other properties
    ReservedConcurrentExecutions: 10  # Limit parallel executions

Troubleshooting

Validation error on invoke: If you get a validation error saying "Execution timeout must be within the function timeout," you're trying to synchronously invoke a function where the execution timeout exceeds the function timeout. Use asynchronous invocation for long-running workflows.

Individual invocation timeout: If your function times out during a single invocation, consider increasing the Timeout property. Start by doubling the current value, up to the maximum of 900 seconds. This is separate from the workflow's total ExecutionTimeout.

Workflow timeout: If your entire workflow times out, consider increasing the ExecutionTimeout in DurableConfig. This can be up to 1 year (31,536,000 seconds).

Permission denied for callbacks: Ensure client functions have the correct IAM permissions to interact with durable executions. Functions sending callbacks need SendDurableExecutionCallback* permissions with the target durable function as the resource.

Summary

Deploying durable functions with AWS SAM involves several key configurations: enable DurableConfig on your function to set execution timeout and retention period, install the durable execution SDK as a dependency, and configure esbuild for TypeScript compilation. AWS SAM facilitates the IAM permissions needed for durable execution operations. For client functions that interact with durable executions, consider explicitly configuring permissions for callback and monitoring operations.

Use Globals to reduce repetition across functions, set appropriate timeouts for both individual invocations and total workflow duration, and enable tracing for observability. Test locally with sam local invoke and sam local execution commands before deploying to AWS.

With these patterns in place, you can deploy long-running workflows that span hours or days, handle failures gracefully, and provide full visibility into execution state.

The Replay Model: How AWS Lambda Durable Functions Actually Work

Eric D Johnson — Wed, 03 Dec 2025 00:27:54 +0000

Understanding the checkpoint-based execution that makes long-running workflows possible

You write an AWS Lambda function that looks like it runs continuously for hours. But Lambda functions can only run for 15 minutes. How does this work?

The answer is replay - a checkpoint-based execution model that makes your function restart from the beginning on every invocation, but skip the work it's already done. It's elegant, efficient, and once you understand it, surprisingly intuitive.

The Core Principle

Here's the fundamental truth about durable functions:

Your handler function re-executes from the beginning on every invocation, but completed operations return cached results from checkpoints instead of re-executing.

Let's see this in action with a simple workflow:

async function processOrder(event: any, ctx: DurableContext) {
  const order = await ctx.step('create-order', async () => {
    console.log('Creating order...');
    return { orderId: '123', total: 50 };
  });

  const payment = await ctx.step('process-payment', async () => {
    console.log('Processing payment...');
    return { transactionId: 'txn-456', status: 'success' };
  });

  await ctx.wait({ seconds: 300 }); // Wait 5 minutes

  const notification = await ctx.step('send-notification', async () => {
    console.log('Sending notification...');
    return { sent: true };
  });

  return { order, payment, notification };
}

Here's what actually happens across two separate Lambda invocations:

Invocation 1 (t=0s):

Creating order...
Processing payment...
[Checkpoint: create-order completed]
[Checkpoint: process-payment completed]
[Function terminates - waiting 5 minutes]

Invocation 2 (t=300s, after wait completes):

[REPLAY MODE: Skipping create-order - returning cached result]
[REPLAY MODE: Skipping process-payment - returning cached result]
[EXECUTION MODE: Running send-notification]
Sending notification...
[Checkpoint: send-notification completed]
[Function completes]

Notice what happened: The function started from the beginning both times, but on the second invocation, create-order and process-payment didn't re-execute. The logs only appeared once, even though the code ran twice. The function seamlessly continued from where it left off.

This is replay.

Execution Modes: The Secret Sauce

The SDK operates in two modes that automatically switch based on what's happening.

ExecutionMode is when the function is executing operations for the first time. Operations execute normally, results are saved to checkpoints, logs are emitted, and side effects happen.

ReplayMode is when the function is replaying previously completed operations. Operations return cached results instantly without actual execution, logs are suppressed, and no side effects occur.

The SDK automatically transitions from ReplayMode to ExecutionMode when it reaches an operation that hasn't been completed yet.

How Checkpoints Work

Every operation creates a checkpoint that stores:

{
  operationId: "2",           // Sequential ID
  operationType: "STEP",      // STEP, WAIT, INVOKE, etc.
  operationName: "process-payment",
  status: "SUCCEEDED",        // STARTED, SUCCEEDED, FAILED, PENDING
  result: {                   // The actual return value
    transactionId: "txn-456",
    status: "success"
  }
}

When your function restarts, the SDK loads all checkpoints from storage, indexes them by operation ID, returns cached results for completed operations, and executes new operations normally.

The Determinism Requirement

For replay to work, your code must be deterministic - the same sequence of operations must happen in the same order every time.

What Breaks Determinism

// ❌ Random control flow
if (Math.random() > 0.5) {
  await ctx.step('optional-step', async () => doSomething());
}
// First run: random = 0.7, step executes, checkpoint created
// Second run: random = 0.3, step skipped
// Error: Expected operation 'optional-step' at position 2, not found!

// ❌ Time-based branching
const isWeekend = new Date().getDay() >= 5;
if (isWeekend) {
  await ctx.step('weekend-task', async () => doWeekendWork());
}
// First run: Friday (day 5), step executes
// Second run: Monday (day 1), step skipped
// Error: Replay consistency violation!

// ❌ External state
let counter = 0;
await ctx.step('step1', async () => {
  counter++; // Won't increment during replay!
  return counter;
});

How to Write Deterministic Code

The rule is simple: capture non-deterministic values inside steps.

// ✅ Capture random values in steps
const randomId = await ctx.step('generate-id', async () => {
  return crypto.randomUUID(); // Executed once, cached on replay
});

// ✅ Capture timestamps in steps
const timestamp = await ctx.step('get-timestamp', async () => {
  return Date.now(); // Same timestamp on every replay
});

// ✅ Use event data for control flow
if (event.shouldProcess) { // Deterministic - same event every time
  await ctx.step('process', async () => doWork());
}

// ✅ Capture time-based decisions in steps
const isWeekend = await ctx.step('check-day', async () => {
  return new Date().getDay() >= 5;
});
if (isWeekend) {
  await ctx.step('weekend-task', async () => doWeekendWork());
}

Replay Consistency Validation

The SDK validates that operations occur in the same order on every invocation:

// What gets validated:
// 1. Operation type (STEP, WAIT, INVOKE)
// 2. Operation name (your identifier)
// 3. Operation position (sequential order)

// Example validation error:
// "Replay consistency violation: Expected operation 'process-payment' 
//  of type STEP at position 2, but found operation 'send-email' of type STEP"

This catches bugs early. If your code's execution path changes between invocations, you'll know immediately.

A Complete Example: Order Processing with Replay

Let's examine a potential workflow through multiple invocations:

async function processOrder(event: any, ctx: DurableContext) {
  ctx.logger.info('Order processing started', { orderId: event.orderId });

  // Step 1: Validate inventory
  const inventory = await ctx.step('check-inventory', async () => {
    ctx.logger.info('Checking inventory');
    const response = await fetch(`https://api.inventory.com/check`, {
      method: 'POST',
      body: JSON.stringify({ items: event.items })
    });
    return response.json();
  });

  if (!inventory.available) {
    ctx.logger.warn('Out of stock', { missing: inventory.missing });
    return { status: 'out-of-stock' };
  }

  // Step 2: Process payment
  const payment = await ctx.step('process-payment', async () => {
    ctx.logger.info('Processing payment', { amount: inventory.total });
    const response = await fetch(`https://api.payments.com/charge`, {
      method: 'POST',
      body: JSON.stringify({ 
        customerId: event.customerId, 
        amount: inventory.total 
      })
    });
    return response.json();
  });

  // Step 3: Wait for warehouse confirmation (5 minute timeout)
  ctx.logger.info('Waiting for warehouse confirmation');
  const confirmation = await ctx.waitForCallback(
    'warehouse-confirm',
    async (callbackId) => {
      // Send callback ID to warehouse system
      await fetch(`https://api.warehouse.com/notify`, {
        method: 'POST',
        body: JSON.stringify({ orderId: order.id, callbackId })
      });
    },
    { timeout: { seconds: 300 } }
  );

  // Step 4: Send notification
  await ctx.step('notify-customer', async () => {
    ctx.logger.info('Sending customer notification');
    await fetch(`https://api.notifications.com/send`, {
      method: 'POST',
      body: JSON.stringify({
        customerId: event.customerId,
        message: 'Your order is confirmed!'
      })
    });
  });

  ctx.logger.info('Order processing completed');
  return { status: 'completed', orderId: payment.orderId };
}

Invocation Timeline

Invocation 1 (t=0s) runs in ExecutionMode. The logs show "Order processing started", "Checking inventory", "Processing payment", and "Waiting for warehouse confirmation". Checkpoints are created for check-inventory and process-payment, both marked as SUCCEEDED. The function then enters a waiting state for the callback 'warehouse-confirm', creating a checkpoint to persist the callbackId and set the timer for the timeout.

Invocation 2 (t=120s, warehouse confirms) starts in ReplayMode and transitions to ExecutionMode. During the replay phase, "Order processing started", "Checking inventory", "Processing payment", and "Waiting for warehouse confirmation" are all suppressed - the inventory and payment steps return their cached results without re-executing. Once the function reaches new operations, it switches to ExecutionMode, checkpoints the callback result, and logs "Sending customer notification" and "Order processing completed". A checkpoint is created for notify-customer marked as SUCCEEDED, and the function completes.

Notice how the function ran from the beginning both times, but the inventory and payment APIs were only called once. Logs only appeared once with no duplicates, and the function seamlessly continued after the callback.

Operation IDs: The Replay Index

Operations are identified by sequential IDs that determine replay order:

// Root context operations
await ctx.step('step1', ...);  // ID: "1"
await ctx.step('step2', ...);  // ID: "2"
await ctx.step('step3', ...);  // ID: "3"

// Child context operations (from ctx.runInChildContext)
await ctx.runInChildContext(async (childCtx) => {  // This operation gets ID: "4"
  await childCtx.step('child-step1', ...);  // ID: "4-1"
  await childCtx.step('child-step2', ...);  // ID: "4-2"
});

IDs are deterministic - they're based on execution order, not operation names. This is why operation order must be consistent.

Common Pitfalls and Solutions

1. Non-Deterministic Control Flow

// ❌ BAD: Random branching
if (Math.random() > 0.5) {
  await ctx.step('optional', async () => doWork());
}

// ✅ GOOD: Event-based branching
if (event.shouldDoWork) {
  await ctx.step('optional', async () => doWork());
}

// ✅ GOOD: Capture decision in step
const shouldDoWork = await ctx.step('decide', async () => {
  return Math.random() > 0.5;
});
if (shouldDoWork) {
  await ctx.step('optional', async () => doWork());
}

2. Mutating Closure Variables

// ❌ BAD: External mutation
let total = 0;
await ctx.step('add-items', async () => {
  total += 10; // Won't happen during replay!
});

// ✅ GOOD: Return values
const total = await ctx.step('calculate-total', async () => {
  return 10;
});

3. Side Effects Outside Steps

// ❌ BAD: Direct API calls
const data = await fetch('https://api.example.com/data');
await ctx.step('process', async () => processData(data));
// API called on every replay!

// ✅ GOOD: API calls inside steps
const data = await ctx.step('fetch-data', async () => {
  return await fetch('https://api.example.com/data');
});
await ctx.step('process', async () => processData(data));

Debugging Replay Issues

When replay goes wrong, use the execution history:

# View execution history
sam remote execution history $EXECUTION_ARN

# See detailed operation data
sam remote execution history $EXECUTION_ARN --format json

The history shows every operation that executed, the order they ran in, their results or errors, and when mode transitions occurred. Look for operations appearing in different orders, missing or extra operations, or operations with different names at the same position.

Best Practices for Replay-Safe Code

Wrap all non-deterministic operations in steps - random numbers, timestamps, API calls, and database queries should always be inside ctx.step(). Use event data for control flow rather than runtime-generated values, and never mutate closure variables - return values from steps instead. Keep operation order consistent so the same sequence happens every time. Test with multiple invocations to verify replay behavior locally, and check execution history to debug replay issues quickly.

Summary

The replay model is what makes durable functions possible. Your function restarts from the beginning on every invocation, but completed operations return cached results without re-executing. The SDK automatically switches between ReplayMode and ExecutionMode, and your code must be deterministic for replay to work correctly.

Once you internalize these principles, writing durable functions becomes natural. You write straightforward procedural code, and the SDK handles all the complexity of checkpointing, replay, and state management. The result? Long-running workflows that look like simple functions. That's the magic of replay.

Testing AWS Lambda Durable Functions in TypeScript

Eric D Johnson — Tue, 02 Dec 2025 23:46:01 +0000

How to write fast, reliable tests for workflows that span hours

Testing long-running workflows is tricky. Your function waits for callbacks, retries failed operations, and spans multiple invocations. How do you test that without actually waiting hours or deploying to AWS?

The AWS Durable Execution SDK includes a testing library that solves this. You can run your durable functions locally, skip time-based waits, inspect every operation, and verify behavior without touching AWS. Let's see how.

The Testing Library

The SDK provides testing tools for different scenarios:

LocalDurableTestRunner runs your function in-process with a simulated checkpoint server. Tests execute in milliseconds, even for workflows that would normally take hours. This is what you'll use for most testing.

CloudDurableTestRunner tests against deployed Lambda functions in AWS. Use this for integration tests or when you need to verify behavior in the actual AWS environment.

run-durable CLI provides quick command-line testing without writing test code. Perfect for rapid iteration and debugging during development.

We'll focus on local testing since that's where you'll spend most of your time.

Your First Test

Let's start with a simple durable function:

// order-processor.ts
import { DurableContext, withDurableExecution } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const order = await context.step('create-order', async () => {
      return { orderId: '123', total: 50 };
    });

    await context.wait({ seconds: 300 }); // Wait 5 minutes

    const notification = await context.step('send-notification', async () => {
      return { sent: true };
    });

    return { order, notification };
  }
);

Here's how to test it:

// order-processor.test.ts
import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';
import { handler } from './order-processor';

describe('Order Processor', () => {
  beforeAll(async () => {
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
  });

  it('should process order successfully', async () => {
    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    const execution = await runner.run();

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()).toEqual({
      order: { orderId: '123', total: 50 },
      notification: { sent: true }
    });
  });
});

That's it. The test runs in milliseconds, even though the function has a 5-minute wait. The skipTime: true option tells the test runner to skip over time-based operations instantly.

Setup and Teardown

Every test suite needs setup and teardown:

beforeAll(async () => {
  await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
});

afterAll(async () => {
  await LocalDurableTestRunner.teardownTestEnvironment();
});

setupTestEnvironment() starts a local checkpoint server and optionally installs fake timers for time skipping. teardownTestEnvironment() cleans everything up. Call these once per test file in beforeAll and afterAll.

The skipTime option is crucial for fast tests. When enabled, operations like context.wait(), setTimeout, and retry delays complete instantly. Without it, your tests would actually wait for the specified durations.

Inspecting Operations

The real power of the testing library is inspecting what your function did:

it('should execute operations in correct order', async () => {
  const runner = new LocalDurableTestRunner({
    handlerFunction: handler,
  });

  const execution = await runner.run();

  // Get all operations
  const operations = execution.getOperations();
  expect(operations).toHaveLength(3); // create-order, wait, send-notification

  // Get specific operation by index
  const createOrder = runner.getOperationByIndex(0);
  expect(createOrder.getType()).toBe('STEP');
  expect(createOrder.getStatus()).toBe('SUCCEEDED');
  expect(createOrder.getStepDetails()?.result).toEqual({
    orderId: '123',
    total: 50
  });

  // Get wait operation
  const waitOp = runner.getOperationByIndex(1);
  expect(waitOp.getType()).toBe('WAIT');
  expect(waitOp.getWaitDetails()?.waitSeconds).toBe(300);

  // Get notification operation
  const notification = runner.getOperationByIndex(2);
  expect(notification.getStepDetails()?.result).toEqual({ sent: true });
});

You can access operations by index, by name, or by ID. Each operation exposes its type, status, and type-specific details like step results or wait durations.

Testing Failures and Retries

Real workflows fail. Let's test retry behavior:

// payment-processor.ts
import { DurableContext, withDurableExecution, createRetryStrategy } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const payment = await context.step('process-payment', async () => {
      const response = await fetch('https://api.payments.com/charge', {
        method: 'POST',
        body: JSON.stringify({ amount: event.amount })
      });
      if (!response.ok) throw new Error('Payment failed');
      return response.json();
    }, {
      retryStrategy: createRetryStrategy({
        maxAttempts: 3,
        backoffRate: 2,
        initialInterval: 1000
      })
    });

    return payment;
  }
);

Test it with mocks:

// payment-processor.test.ts
import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';
import { handler } from './payment-processor';

// Store original fetch
const originalFetch = global.fetch;

describe('Payment Processor', () => {
  beforeAll(async () => {
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
  });

  beforeEach(() => {
    // Mock fetch for external APIs only
    global.fetch = jest.fn((url: string | URL | Request, ...args) => {
      const urlString = url.toString();
      // Let checkpoint server calls through
      if (urlString.includes('127.0.0.1') || urlString.includes('localhost')) {
        return originalFetch(url as any, ...args);
      }
      // Mock external API calls
      return Promise.reject(new Error('Unmocked fetch call'));
    }) as any;
  });

  afterEach(() => {
    global.fetch = originalFetch;
  });

  it('should succeed on first attempt', async () => {
    (global.fetch as jest.Mock).mockImplementation((url: string | URL | Request, ...args) => {
      const urlString = url.toString();
      if (urlString.includes('127.0.0.1') || urlString.includes('localhost')) {
        return originalFetch(url as any, ...args);
      }
      if (urlString.includes('api.payments.com')) {
        return Promise.resolve({
          ok: true,
          json: async () => ({ transactionId: 'txn-123', status: 'success' })
        });
      }
      return Promise.reject(new Error('Unmocked fetch'));
    });

    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    const execution = await runner.run({ payload: { amount: 100 } });

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()).toEqual({
      transactionId: 'txn-123',
      status: 'success'
    });
  });

  it('should retry on failure and eventually succeed', async () => {
    let callCount = 0;
    (global.fetch as jest.Mock).mockImplementation((url: string | URL | Request, ...args) => {
      const urlString = url.toString();
      if (urlString.includes('127.0.0.1') || urlString.includes('localhost')) {
        return originalFetch(url as any, ...args);
      }
      if (urlString.includes('api.payments.com')) {
        callCount++;
        if (callCount === 1) {
          return Promise.reject(new Error('Network error'));
        }
        return Promise.resolve({
          ok: true,
          json: async () => ({ transactionId: 'txn-456', status: 'success' })
        });
      }
      return Promise.reject(new Error('Unmocked fetch'));
    });

    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    const execution = await runner.run({ payload: { amount: 100 } });

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()).toEqual({
      transactionId: 'txn-456',
      status: 'success'
    });
  });

  it('should fail after exhausting retries', async () => {
    (global.fetch as jest.Mock).mockImplementation((url: string | URL | Request, ...args) => {
      const urlString = url.toString();
      if (urlString.includes('127.0.0.1') || urlString.includes('localhost')) {
        return originalFetch(url as any, ...args);
      }
      if (urlString.includes('api.payments.com')) {
        return Promise.reject(new Error('Persistent failure'));
      }
      return Promise.reject(new Error('Unmocked fetch'));
    });

    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    const execution = await runner.run({ payload: { amount: 100 } });

    expect(execution.getStatus()).toBe('FAILED');
    const error = execution.getError();
    expect(error?.errorMessage).toBe('Persistent failure');
    expect(error?.errorType).toBe('StepError');
  });
});

With skipTime: true, the retry delays happen instantly. The test verifies retry behavior without actually waiting for backoff intervals.

Important: When mocking fetch, preserve calls to localhost/127.0.0.1 so the checkpoint server can communicate. Only mock external API calls.

Testing Callbacks

Durable functions often wait for external events. Here's how to test that:

// approval-workflow.ts
import { DurableContext, withDurableExecution } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const request = await context.step('create-request', async () => {
      return { requestId: event.requestId, status: 'pending' };
    });

    // Wait up to 24 hours for approval
    try {
      const approval = await context.waitForCallback('approval', 86400);
      return { status: 'approved', approvedBy: approval.userId };
    } catch (error) {
      return { status: 'timeout' };
    }
  }
);

Test with callback simulation:

// approval-workflow.test.ts
import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';
import { handler } from './approval-workflow';

describe('Approval Workflow', () => {
  beforeAll(async () => {
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
  });

  it('should complete when approval is received', async () => {
    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    const execution = runner.run({ payload: { requestId: '123' } });

    // Get the callback operation
    const callbackOp = runner.getOperation('approval');

    // Simulate approval callback
    await callbackOp.sendCallbackSuccess({ userId: 'user-456' });

    const result = await execution;

    expect(result.getStatus()).toBe('SUCCEEDED');
    expect(result.getResult()).toEqual({
      status: 'approved',
      approvedBy: 'user-456'
    });
  });

  it('should timeout when no approval is received', async () => {
    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    // Don't send callback - let it timeout
    const execution = await runner.run({ payload: { requestId: '123' } });

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()).toEqual({ status: 'timeout' });
  });
});

The getOperation() method returns an operation handle that lets you send callbacks using sendCallbackSuccess(). With skipTime: true, the timeout happens instantly if no callback is sent.

Testing Parallel Operations

Durable functions can run operations in parallel:

// batch-processor.ts
import { DurableContext, withDurableExecution } from '@aws/durable-execution-sdk-js';

export const handler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    await context.parallel(
      'process-items',
      event.items.map((item: any, index: number) =>
        async (childContext: DurableContext) => {
          return await childContext.step(`process-item-${index}`, async () => {
            return { id: item.id, processed: true };
          });
        }
      )
    );

    return {
      total: event.items.length,
      successful: event.items.length
    };
  }
);

Test parallel execution:

// batch-processor.test.ts
import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';
import { handler } from './batch-processor';

describe('Batch Processor', () => {
  beforeAll(async () => {
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
  });

  it('should process all items in parallel', async () => {
    const runner = new LocalDurableTestRunner({
      handlerFunction: handler,
    });

    const execution = await runner.run({
      payload: {
        items: [
          { id: '1' },
          { id: '2' },
          { id: '3' }
        ]
      }
    });

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()).toEqual({
      total: 3,
      successful: 3
    });

    // Verify operations were tracked
    const operations = execution.getOperations();
    expect(operations.length).toBeGreaterThan(0);
  });
});

The context.parallel() method takes a name and an array of functions that each receive a child context. Each parallel operation is tracked independently.

Testing Nested Functions

Durable functions can invoke other durable functions:

// main-workflow.ts
import { DurableContext, withDurableExecution } from '@aws/durable-execution-sdk-js';

export const childHandler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const result = await context.step('child-step', async () => {
      return { processed: event.data };
    });
    return result;
  }
);

export const mainHandler = withDurableExecution(
  async (event: any, context: DurableContext) => {
    const childResult = await context.invoke('child-function', {
      data: event.input
    });

    return { main: 'completed', child: childResult };
  }
);

// main-workflow.test.ts
import { LocalDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';
import { mainHandler, childHandler } from './main-workflow';

describe('Main Workflow', () => {
  beforeAll(async () => {
    await LocalDurableTestRunner.setupTestEnvironment({ skipTime: true });
  });

  afterAll(async () => {
    await LocalDurableTestRunner.teardownTestEnvironment();
  });

  it('should invoke child function successfully', async () => {
    const runner = new LocalDurableTestRunner({
      handlerFunction: mainHandler,
    });

    // Register the child function
    runner.registerDurableFunction('child-function', childHandler);

    const execution = await runner.run({ payload: { input: 'test-data' } });

    expect(execution.getStatus()).toBe('SUCCEEDED');
    expect(execution.getResult()).toEqual({
      main: 'completed',
      child: { processed: 'test-data' }
    });
  });
});

The registerDurableFunction() method tells the test runner how to handle context.invoke() calls. Without registration, invocations would fail.

Quick Testing with the CLI

For rapid iteration without writing test code, use the run-durable CLI:

npm run run-durable -- path/to/your-function.ts

The CLI runs your function locally and displays an operations table automatically. Useful options:

no-skip-time - Actually wait for delays (default skips time)
verbose - Show detailed execution logs
show-history - Display history events table

# Run with verbose logging and history
npm run run-durable -- src/order-processor.ts verbose show-history

The CLI is perfect for quick debugging and verifying function behavior without setting up a full test suite.

Debugging with Execution History

When tests fail, the execution history helps you understand what happened:

it('should provide detailed execution history', async () => {
  const runner = new LocalDurableTestRunner({
    handlerFunction: handler,
  });

  const execution = await runner.run();

  // Print operations table to console
  execution.print();

  // Get detailed history events
  const history = execution.getHistoryEvents();
  console.log('History:', JSON.stringify(history, null, 2));

  // Get all invocations
  const invocations = execution.getInvocations();
  console.log('Invocations:', invocations.length);

  // Filter operations by status
  const succeeded = execution.getOperations({ status: 'SUCCEEDED' });
  const failed = execution.getOperations({ status: 'FAILED' });
});

The print() method outputs a formatted table of all operations. The history events show every checkpoint, invocation, and state transition. You can filter operations by status to focus on specific outcomes.

Additional operation access methods:

getOperationByNameAndIndex(name, index) - Get specific occurrence of a named operation
getOperationById(id) - Get operation by its unique ID

Testing Without Time Skipping

Sometimes you want to test actual timing behavior:

it('should respect actual wait durations', async () => {
  // Setup without skipTime
  await LocalDurableTestRunner.setupTestEnvironment({ skipTime: false });

  const runner = new LocalDurableTestRunner({
    handlerFunction: handler,
  });

  const startTime = Date.now();
  await runner.run();
  const duration = Date.now() - startTime;

  // Should have actually waited ~2 seconds
  expect(duration).toBeGreaterThanOrEqual(2000);

  await LocalDurableTestRunner.teardownTestEnvironment();
});

Without skipTime, waits and retries happen in real time. Use this sparingly - most tests should skip time for speed.

Cloud Testing

For integration tests against deployed functions:

import { CloudDurableTestRunner } from '@aws/durable-execution-sdk-js-testing';

describe('Deployed Function Integration Tests', () => {
  it('should execute in AWS', async () => {
    const runner = new CloudDurableTestRunner({
      functionName: 'OrderProcessorFunction:$LATEST',
      region: 'us-east-1',
    });

    const execution = await runner.run({ payload: {} });

    expect(execution.getStatus()).toBe('SUCCEEDED');
  }, 600000); // 10 minute timeout for cloud tests
});

Cloud testing invokes your actual Lambda function and waits for completion. Important notes:

Qualifier required: You must specify a qualifier (:$LATEST, :prod, etc.) when invoking durable functions
Execution timeout constraint: For synchronous invocation, the function's ExecutionTimeout in DurableConfig must not exceed 900 seconds (15 minutes)
Test timeout: Set a Jest timeout that accounts for your workflow duration - cloud tests run in real time
Real waits: Unlike local tests, cloud tests actually wait for the specified durations (e.g., 5 minutes for a 300-second wait)
AWS credentials: Ensure AWS credentials are configured. You can set the AWS_PROFILE environment variable or use other AWS credential providers

Cloud testing is slower than local testing but verifies behavior in the real AWS environment with actual checkpointing and state management.

Best Practices

Use local testing for most tests. It's fast, doesn't require AWS credentials, and gives you full control over timing and callbacks.

Skip time by default. Set skipTime: true in setupTestEnvironment() unless you specifically need to test timing behavior.

Mock external dependencies. Use Jest mocks for API calls, database operations, and other external services. This keeps tests fast and deterministic.

Test failure scenarios. Don't just test the happy path. Verify retry behavior, timeout handling, and error propagation.

Inspect operations, not just results. Use getOperation() and getOperations() to verify that your function executed the right steps in the right order.

Register nested functions. If your function uses context.invoke(), register the child functions with registerDurableFunction().

Use cloud testing sparingly. Reserve CloudDurableTestRunner for integration tests that need to verify behavior in AWS.

Common Patterns

Testing with dynamic operation names:

it('should handle dynamic operation names', async () => {
  const runner = new LocalDurableTestRunner({
    handlerFunction: handler,
  });

  const execution = await runner.run({ payload: { userId: '123' } });

  // Get operation by name pattern
  const userOp = runner.getOperation('process-user-123');
  expect(userOp.getStatus()).toBe('SUCCEEDED');
});

Testing error propagation:

it('should propagate errors correctly', async () => {
  const runner = new LocalDurableTestRunner({
    handlerFunction: handler,
  });

  const execution = await runner.run();

  expect(execution.getStatus()).toBe('FAILED');
  const error = execution.getError();
  expect(error?.errorType).toBe('StepError');
  expect(error?.errorMessage).toContain('expected error message');
});

Testing with multiple callbacks:

it('should handle multiple callbacks', async () => {
  const runner = new LocalDurableTestRunner({
    handlerFunction: handler,
  });

  const execution = runner.run();

  const callback1 = runner.getOperation('callback-1');
  const callback2 = runner.getOperation('callback-2');

  await callback1.sendCallback({ data: 'first' });
  await callback2.sendCallback({ data: 'second' });

  const result = await execution;
  expect(result.getStatus()).toBe('SUCCEEDED');
});

Summary

The Durable Execution testing library makes it easy to test long-running workflows. Run functions locally with simulated checkpoints, skip time-based operations for fast tests, inspect every operation, and verify behavior without deploying to AWS.

Use LocalDurableTestRunner for fast unit tests with full control over timing and callbacks. Use CloudDurableTestRunner for integration tests against deployed functions. Mock external dependencies, test failure scenarios, and inspect operations to verify correct behavior.

With these tools, testing durable functions is as straightforward as testing any other code. Write tests first, iterate quickly, and ship workflows with confidence.

Forem: Eric D Johnson

Modern Container Builds and WebSocket APIs Come to AWS SAM

syntax=docker/dockerfile:1

Need v1.156.0 or later for BuildKit, latest for WebSocket APIs

Upgrade via pip

Or via Homebrew

Cold Starts Are Dead

How Long Are Lambda Cold Starts in 2026?

Does VPC Still Cause Lambda Cold Starts?

How Much Does SnapStart Reduce Cold Starts?

Did the Lambda INIT Billing Change Increase Costs?

I Decided to See for Myself

When Do Lambda Cold Starts Still Matter?

Conclusion

For the Agents

Is this code deterministic?

Problem 1: Equal-priority ordering is under-specified

What’s happening

Why it still matters (and why this one is subtle)

Why not just wrap the ordering in a step?

Fix

Problem 2: Date.now() is non-deterministic

What’s happening

Why it matters

Fix

Problem 3: Side effects hidden inside a single step

What’s happening

Why it matters

Fix

Step semantics and retry intent

Putting it all together

Durable-safe handler (step by step)

Durable-safe handler using context.map()

Final takeaway

AWS Lambda Durable Functions: Build Workflows That Last

What Are Durable Functions?

A Simple Example

Key Capabilities

How It Works: The Replay Model

Common Use Cases

Testing Your Workflows

Deploying with AWS SAM

When to Use Durable Functions

Getting Started

Learn More

Summary

Developing AWS Lambda Durable Functions with AWS SAM

Prerequisites

The Basics: Enabling Durable Functions

Installing the SDK

Build Configuration

IAM Permissions

Using Globals for Common Configuration

A Complete Example

Building and Deploying

Local Testing

Accessing Cloud Resources Locally

Tracking Execution State

Remote Testing

Configuration Best Practices

Monitoring and Observability

Common Patterns

Troubleshooting

Summary

The Replay Model: How AWS Lambda Durable Functions Actually Work

The Core Principle

Execution Modes: The Secret Sauce

How Checkpoints Work

The Determinism Requirement

What Breaks Determinism

How to Write Deterministic Code

Replay Consistency Validation

A Complete Example: Order Processing with Replay

Invocation Timeline

Operation IDs: The Replay Index

Common Pitfalls and Solutions

1. Non-Deterministic Control Flow

2. Mutating Closure Variables

3. Side Effects Outside Steps

Debugging Replay Issues

Problem 2: `Date.now()` is non-deterministic

Durable-safe handler using `context.map()`