Forem: Daniele Frasca

Three Responsibilities of a Global Application (Part 2)

Daniele Frasca — Tue, 17 Mar 2026 05:00:00 +0000

In Part 1, I explained that at a global scale, trust is part of the architecture. Not trust as a feeling, but trust as something the system must enforce and prove.

In this article, I aim to explain the 3 distinct responsibilities that enable the system to grow organically.

Why global systems feel complex

Most complexity in global systems does notcome from services. It comes from mixing concerns.
I have seen this pattern many times:

CloudWatch dashboards are used to answer audit questions
CloudTrail logs are pulled into debugging workflows
Metrics start carrying tenant identifiers just to be safe

None of the above are wrong in isolation, but when they are together, they create systems that are:

Hard to operate
Hard to explain
Hard to defend

The problem is the system is trying to answer too many different questions at once. In short, it is the same pattern that applies to code Single Responsibility.
My job brought me to a point where I stopped thinking in terms of architectures and started thinking in terms of responsibilities. No matter how the application is built, it must answer the same 3 questions:

How does work actually happen?
How can we prove that work happened correctly?
How do we know whether the system is healthy?

When these responsibilities are clearly separated, decisions become easier. When they are mixed, every discussion becomes a mess.

Responsibility #1 — Doing the work (execution)

This is the responsibility most devs are comfortable with.

It is where:

Business logic runs
Requests are processed
Events are handled
Workflows progress

In AWS terms, this is:

AWS Lambda
Step Functions
EventBridge
DynamoDB
SQS
SNS

This responsibility answers one question only:

What does the system do for the business?

And it should be optimised for:

Correctness
Scalability
Resilience
Isolation

Problems start when this responsibility is overloaded.
Examples:

embedding compliance logic directly into business code
adding just in case logging everywhere without structure
leaking operational concerns into domain logic

Execution code should focus on doing the work, not explaining or defending it.

Responsibility #2 — Proving the work (evidence and control)

This responsibility exists because someone outside the team will ask questions like:

Who had access?
Who changed production?
What data moved where?
Was logging enabled at the time?

This responsibility is not about debugging. It is about proof.

In AWS, this responsibility is expressed through things like:

AWS CloudTrail
IAM configuration and access records
Configuration history
Retention policies
AWS Audit Manager

A common issue is teams trying to reuse execution or observability data as evidence. That usually fails because:

Logs change format
Metrics are aggregated
Dashboards get deleted
Devs remember things differently

Evidence systems must be:

Complete
Consistent
Tamper‑resistant

This is why this responsibility must be separate from execution.

Responsibility #3 — Understanding the system (operations)

This responsibility answers a very different question:

Is the system healthy right now?

Not:

What happened to tenant X?
Who changed this?

But:

Are errors increasing?
Is latency degrading?
Is this regional or global?

And the answers are:

Metrics
Alerts
SLOs
Dashboards

In AWS environments, this usually means:

CloudWatch metrics
Amazon Managed Prometheus
Service telemetry
Alerts

Those services exist, they trigger the investigation and actions.

Why mixing responsibilities breaks systems

Once these 3 responsibilities are separated, many things become obvious. For example:

Metrics with tenantId - That is usually execution detail leaking into operations, and this is not what metrics are actually meant for.
CloudWatch dashboards as audit - Dashboards explain system behaviour while auditors need immutable, verifiable evidence.
Debugging incidents by scrolling through CloudTrail - CloudTrail is excellent at answering who did what, but it is a terrible tool for answering why the system is behaving this way right now.

Each of these feels ok on its own, but at scale, there is confusion about what the system is actually trying to tell us.

Benefits of separation

Once responsibilities are separated, conversations change.

Instead of:

Should we centralise logs?

I ask:

Which responsibility are we trying to serve?

Instead of:

Why cannot I just add tenantId to metrics?

I ask:

Is this an operational signal or an accounting question?

Instead of:

Why is governance slowing us down?

I ask:

Which responsibility are we trying to satisfy?

The trade‑offs are the same, but they are made explicit.

Conclusion

From a governance angle, global systems do not fail because they are distributed. They fail because we ask one system to do everything at once.
Separating responsibilities does notreduce complexity, it puts complexity where it belongs.

Trust Is the Architecture (Part 1)

Daniele Frasca — Wed, 04 Mar 2026 05:00:00 +0000

Over the past few years, I have written about serverless from a technical perspective, covering topics like latency, multi-region setups, and complex scenarios. That is all important, but this time I want to shift focus. Instead of building and deploying, I want to talk about what happens next: how to handle governance and answer the question:

How do I run this when the blast radius is large, the org is big, and somebody outside engineering needs proof?

This is Part 1, where I will introduce the main idea and share my perspective.

Looking back at my journey with serverless, I started out using managed services to deliver value and ship features faster. As traffic increased, I learned how concurrency, retries, throttling, and quotas work. That is when I added caching and other solutions to help the system scale. I also discovered the limits of the system the hard way, especially when multiple teams relied on the same things. This is when Serverless stopped being just a way to build. At a global scale the problems are becoming about:

How teams deploy and scale
How failures propagate
How data is accessed

My point of view has shifted from Can we build it? to Can we trust it?

When I say "trust", I am not talking about feelings but how a system operate

Can we trust that a tenant boundary is real?
Can we trust that production changes are controlled?
Can we trust that data does not leave the region?
Can we trust that when something goes wrong, we can explain what happened?
Can we trust that if an auditor asks, we have evidence, not just stories?

I have worked in places where trust is based on hope:

Daniele is careful
We all know not to touch that thing in production
I will fix it if it breaks

That is not bad, but it does not scale. Informal rules get forgotten, people get tired, and mistakes happen all the time. So when I say "Trust," I mean being able to make a claim about how the system works and back it up with evidence.

If I do not rely on people, I start thinking in this way:

"Only approved pipelines can deploy to production."
"EU personal data does not leave EU regions."
"A developer cannot disable logging."
"I can trace an incident end-to-end across services."

This means that "Trust" does not depend on people, but is the set of decisions so that the system itself ensures the right things happen and the wrong things become impossible or extremely hard.

Serverless does not create the need for trust, it just makes it more visible, since the application is split into many separate services. We can find some of the answers by combining:

AWS CloudTrail -> who did what, when
Logs and telemetry -> what happened at runtime
Snapshots -> what was true at a point in time
AWS Audit Manager -> automated evidence

If the application is global and regulated, or simply part of a large company, this becomes a core part of how the app is designed. The goal is to make sure the right evidence exists by default.

When trust fails

When a system feels unsafe at scale, it is usually because of one of these problems:

1) Trust between teams

Goal: teams can ship independently without accidentally breaking each other, and without relying on heroics.

The reality is sometimes different:

Shared components with unclear ownership
Production access via long-lived admin roles
Open a ticket and wait for someone to solve it

That is why ownership, approval, and permissions need to be clear. For example, a good practice is to have:

AWS Organizations + Service Control Policies (SCPs) used as permission guardrails across
Production access with time-bound access patterns and temporary elevated access with TEAM

There is a trade-off: more structure means less freedom at first, but I have learned it leads to faster progress later.

2) Trust between systems - EDA

Goal: systems can interact through events without coupling

The reality is sometimes different:

Events without owners
Schema changes without coordination
Accidentally ingesting sensitive data

This is where the "contracts" exist, and tools like EventCatalog are essential because they explicitly define ownership, contracts, rules, and boundaries.

There is a trade-off here too: more explicit contracts can feel slower at first, but the benefit is that I stop guessing about the cost of decoupling.

3) Trust across boundaries - data residency

Goal: prove where data is stored and processed, and prevent accidental cross-boundary movement.

Multi-region is simple to implement, but the hard part is the compliance:

What data can be stored where
What processing is allowed where
What needs to stay inside the region (logs, audit, telemetry, tax, PII, etc.)

Tools like SaaS Lens treat tenant isolation and deployment models as architectural decisions, not just operational ones. Compliance rules make it harder to centralize, so it is important to plan from day one that "regional by default" is the norm, not the exception.

4) Trust with external - auditors, regulators, customers

Goal: claims are backed up by evidence, quickly, repeatedly, and consistently.

I learned the hard way what evidence really means. For example:

Who had access?
Who changed production?
What data moved where?
Is logging complete and tamper resistant?

Questions like these seem simple, but it is often hard to provide evidence after the application is built. Unless there is a major misconfiguration in toos like CloudTrail and other we still need to live with the reality of:

Logs exist but are inconsistent
Evidence exists but it takes weeks to assemble manually

That is why it is so important to follow best practices and use tools like AWS Audit Manager that help scale evidence collection by gathering and organizing data from AWS services and control sources. Most importantly, teams must follow the same patterns everywhere, making logs, metrics, and events consistent.

Conclusion

If this article sounds boring and focused on governance, that is because it is. When serverless goes global, the hardest challenges are not Lambda timeouts or DynamoDB keys. The challenges are:

making boundaries real (and preventing cross-boundary mistakes)
scaling delivery without central bottlenecks
making event-driven coupling visible and owned
producing evidence on demand

That is why I say trust is part of the architecture. Bureaucracy usually slows things down and adds extra work, but when it is automated and built into daily routines, it reduces confusion and helps prevent unsafe situations. In theory, this lets me move faster without breaking things.

Building long-running usage reports on AWS

Daniele Frasca — Thu, 19 Feb 2026 05:00:00 +0000

In this post, I will discuss how to generate a report that could take a long time, and I will use 2 different implementations to see the differences.

Implementation A: AWS Step Functions (Standard workflows)
Implementation B: AWS Lambda Durable Functions

The real challenge is not the HTTP request per se, but rather when the request is accepted.

When a workflow involves long running operations, I must be able to:

stop compute while waiting
resume execution later
preserve state safely
handle retries and failures without re-running everything

Lambda Durable Functions exist because teams needed a way to pause a Lambda execution while external work (for example, an LLM generating a response) is still running without paying for an idle VM and without losing execution state.

However, this is not a new problem.

AWS Step Functions have supported this model for years using:

explicit state machines
task tokens and callbacks

In this article, I explore the same requirement implemented with both approaches to make the trade offs concrete rather than theoretical.

Note:
The code samples are meant to illustrate orchestration patterns and tradeoffs. Some helper functions are shortened to keep the focus on the main architecture.

The flow
Request contract
CreateOrGetReport Lambda
Athena aggregation
Metrics materialization
Queueing narrative generation
Implementation A: Step Functions
- State machine (ASL)
- Athena resume handler
- SQS worker
Implementation B: Lambda Durable Functions
- Durable orchestrator
- Athena resume handler
- SQS worker
Cost comparison
Conclusion

The flow

The process is the same whether you use Step Functions or Durable Lambda. The difference is where the state machine lives.

POST /reports with { tenantId, period, dimensions, metrics, prompt, styles[] }
CreateOrGetReport Lambda
- validates & normalises metricsSpec
- computes metricsKey
- writes job state to DynamoDB (idempotent)
- starts orchestration (if NEW)
Orchestrator starts Athena query
- registers a wait (event-driven)
- waits for Athena completion via EventBridge
Orchestrator stores metrics.json in S3
- this happens ONCE per metricsKey
For each requested narrative style:
- orchestrator sends ONE SQS message (metricsKey + prompt + style + resume token)
- orchestrator waits for completion (no compute running)
LLM worker
- consumes SQS message
- calls OpenAI
- stores narrative-${style}.json in S3
- signals orchestration to resume
Orchestrator
- marks job DONE in DynamoDB
- returns pointers to artifacts

1) Request contract

The client sends a prompt that describes what to aggregate, but the backend handles the actual execution.

Example request:

{
  "tenantId": "acme",
  "period": "2026-01",
  "prompt": "Generate my usage report for last month and explain anomalies",
  "dimensions": ["endpoint"],
  "metrics": ["totalRequests", "topEndpoints", "anomalies"],
  "styles": [
    { "id": "executive", "prompt": "Write an executive summary." },
    { "id": "technical", "prompt": "Write a technical explanation." },
    { "id": "anomalies", "prompt": "Explain anomalies and likely causes." }
  ]
}

2) CreateOrGetReport Lambda

This Lambda is behind API Gateway and handles three things:

1) Normalise the user’s request into a deterministic metricsSpec

2) Compute a stable dedupe key (metricsKey)

3) Upsert a job item in DynamoDB with a conditional write (idempotent)

The DynamoDB table could look like:

PK: TENANT#{tenantId}
SK: REPORT#{period}#{metricsKey} (or just REPORT#{metricsKey} if period is already inside the key)
Attributes: status, reportS3Prefix, createdAt, updatedAt, etc.

CreateOrGetReport (handler)

import { APIGatewayProxyHandlerV2 } from "aws-lambda";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand, GetCommand } from "@aws-sdk/lib-dynamodb";
import { SFNClient, StartExecutionCommand } from "@aws-sdk/client-sfn";
import crypto from "crypto";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const sfn = new SFNClient({});

const JOBS_TABLE = process.env.JOBS_TABLE!;
const ORCHESTRATOR_ARN = process.env.ORCHESTRATOR_ARN!;

const ALLOWED_DIMENSIONS = ["endpoint", "region", "statusCode"] as const;
const ALLOWED_METRICS = ["totalRequests", "topEndpoints", "anomalies"] as const;

type AllowedDimension = (typeof ALLOWED_DIMENSIONS)[number];
type AllowedMetric = (typeof ALLOWED_METRICS)[number];

type Style = { id: string; prompt: string };

type ReportRequest = {
  tenantId: string;
  period: string; // YYYY-MM
  prompt: string;
  dimensions: AllowedDimension[];
  metrics: AllowedMetric[];
  styles: Style[];
};

type MetricsSpec = {
  tenantId: string;
  period: string;
  dimensions: AllowedDimension[];
  metrics: AllowedMetric[];
};

function canonicalJson(obj: unknown): string {
  // Deterministic JSON (stable keys). For production, use a tested library.
  return JSON.stringify(obj, Object.keys(obj as any).sort());
}

function sha256(s: string): string {
  return crypto.createHash("sha256").update(s).digest("hex");
}

function assertAllowed<T extends readonly string[]>(allowed: T, values: string[], field: string): asserts values is T[number][] {
  for (const v of values) {
    if (!(allowed as readonly string[]).includes(v)) {
      throw new Error(`Invalid ${field}: ${v}`);
    }
  }
}

function normalizeMetricsSpec(input: ReportRequest): MetricsSpec {
  if (!input.tenantId || !input.period) throw new Error("tenantId/period required");
  if (!Array.isArray(input.dimensions) || !Array.isArray(input.metrics)) throw new Error("dimensions/metrics required");

  assertAllowed(ALLOWED_DIMENSIONS, input.dimensions as any, "dimension");
  assertAllowed(ALLOWED_METRICS, input.metrics as any, "metric");

  const dimensions = Array.from(new Set(input.dimensions)).sort() as AllowedDimension[];
  const metrics = Array.from(new Set(input.metrics)).sort() as AllowedMetric[];

  return {
    tenantId: input.tenantId,
    period: input.period,
    dimensions,
    metrics
  };
}

export const handler: APIGatewayProxyHandlerV2 = async (event) => {
  const body = event.body ? JSON.parse(event.body) : {};
  const req = body as ReportRequest;

  const metricsSpec = normalizeMetricsSpec(req);
  const metricsKey = sha256(canonicalJson(metricsSpec));

  const pk = `TENANT#${req.tenantId}`;
  const sk = `REPORT#${req.period}#${metricsKey}`;

  const now = new Date().toISOString();
  const reportS3Prefix = `reports/${req.tenantId}/${req.period}/${metricsKey}/`;

  // Attempt to create job (idempotent) using conditional write.
  try {
    await ddb.send(new PutCommand({
      TableName: JOBS_TABLE,
      Item: {
        pk, sk,
        tenantId: req.tenantId,
        period: req.period,
        metricsKey,
        metricsSpec,
        status: "IN_PROGRESS",
        reportS3Prefix,
        styles: req.styles,
        prompt: req.prompt,
        createdAt: now,
        updatedAt: now
      },
      ConditionExpression: "attribute_not_exists(pk) AND attribute_not_exists(sk)"
    }));

    // NEW job → start orchestration
    await sfn.send(new StartExecutionCommand({
      stateMachineArn: ORCHESTRATOR_ARN,
      input: JSON.stringify({
        pk, sk,
        tenantId: req.tenantId,
        period: req.period,
        metricsKey,
        metricsSpec,
        prompt: req.prompt,
        styles: req.styles,
        reportS3Prefix
      })
    }));

    return {
      statusCode: 202,
      body: JSON.stringify({ status: "IN_PROGRESS", metricsKey, reportS3Prefix })
    };
  } catch (err: any) {
    // ConditionalCheckFailedException → job already exists
    const existing = await ddb.send(new GetCommand({
      TableName: JOBS_TABLE,
      Key: { pk, sk }
    }));

    const item = existing.Item as any;
    const status = item?.status ?? "UNKNOWN";

    // If DONE, return pointers; if IN_PROGRESS, return same report key.
    return {
      statusCode: status === "DONE" ? 200 : 202,
      body: JSON.stringify({
        status,
        metricsKey,
        reportS3Prefix: item?.reportS3Prefix,
        artifacts: item?.artifacts
      })
    };
  }
};

3) Athena aggregation

Assume the events are stored in Parquet format and partitioned like this:

s3://usage/tenant_id=acme/month=2026-01/...

A query could look like:

WITH summary AS (
  SELECT COUNT(*) AS total_requests
  FROM usage_parquet
  WHERE tenant_id = :tenantId
    AND month = :period
),
endpoints AS (
  SELECT endpoint, COUNT(*) AS requests
  FROM usage_parquet
  WHERE tenant_id = :tenantId
    AND month = :period
  GROUP BY endpoint
  ORDER BY requests DESC
  LIMIT 10
)
SELECT
  summary.total_requests,
  ARRAY_AGG(endpoints.endpoint) AS top_endpoints
FROM summary, endpoints;

Athena completion events

Athena can emit query state change events to EventBridge:

https://docs.aws.amazon.com/athena/latest/ug/athena-events.html

I will use it to resume orchestration.

4) Metrics materialisation

Athena produces a result once for each metricsKey and stores it as:

metrics.json (S3)
status + pointers (DynamoDB)

Example metrics JSON:

{
  "tenantId": "acme",
  "period": "2026-01",
  "totalRequests": 18234567,
  "topEndpoints": ["POST /ads", "GET /reports"],
  "anomalies": [{ "day": "2026-01-12", "delta": "+320%" }]
}

S3 layout:

s3://my-report-bucket/
  reports/acme/2026-01/<metricsKey>/
    metrics.json
    narrative-executive.json
    narrative-technical.json
    narrative-anomalies.json

5) Queueing narrative generation

OpenAI could limit the requests:

rate limits (429)
transient errors (5xx)
sensitivity to bursts (e.g., month-end report spikes)

And so we need to control this part just in case, using the callback pattern:

Step Functions → task token pattern (worker calls SendTaskSuccess/Failure)
Durable Lambda → callback pattern (worker calls SendDurableExecutionCallbackSuccess/Failure)

Implementation A: Step Functions

We’ll use:

Step Functions Standard workflow
Event driven resume for Athena via EventBridge + a resume Lambda
SQS .waitForTaskToken for narrative workers

Docs

Step Functions task token/callback pattern: https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html
Step Functions error handling and retries: https://docs.aws.amazon.com/step-functions/latest/dg/concepts-error-handling.html
Step Functions + Athena service integration (note: callback pattern is not supported directly for Athena): https://docs.aws.amazon.com/step-functions/latest/dg/connect-athena.html

A1) Step Functions state machine

{
  "Comment": "Usage report orchestration (Athena + queued narratives)",
  "StartAt": "StartAthena",
  "States": {
    "StartAthena": {
      "Type": "Task",
      "Resource": "arn:aws:states:::athena:startQueryExecution",
      "Parameters": {
        "QueryString.$": "$.athena.queryString",
        "QueryExecutionContext": { "Database.$": "$.athena.database" },
        "ResultConfiguration": { "OutputLocation.$": "$.athena.outputLocation" }
      },
      "ResultPath": "$.athenaStart",
      "Retry": [
        {
          "ErrorEquals": ["Athena.TooManyRequestsException", "Athena.InternalServerException"],
          "IntervalSeconds": 2,
          "BackoffRate": 2,
          "MaxAttempts": 6
        }
      ],
      "Next": "RegisterAndWaitForAthena"
    },

    "RegisterAndWaitForAthena": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
      "Parameters": {
        "FunctionName": "${RegisterAthenaWaitLambdaArn}",
        "Payload": {
          "taskToken.$": "$$.Task.Token",
          "queryExecutionId.$": "$.athenaStart.QueryExecutionId",
          "pk.$": "$.pk",
          "sk.$": "$.sk"
        }
      },
      "TimeoutSeconds": 86400,
      "ResultPath": "$.athenaResult",
      "Next": "PutMetricsToS3"
    },

    "PutMetricsToS3": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${WriteMetricsLambdaArn}",
        "Payload": {
          "reportS3Prefix.$": "$.reportS3Prefix",
          "metrics.$": "$.athenaResult.metrics"
        }
      },
      "ResultPath": "$.metricsWrite",
      "Next": "NarrativesMap"
    },

    "NarrativesMap": {
      "Type": "Map",
      "ItemsPath": "$.styles",
      "MaxConcurrency": 10,
      "Parameters": {
        "tenantId.$": "$.tenantId",
        "period.$": "$.period",
        "metricsKey.$": "$.metricsKey",
        "reportS3Prefix.$": "$.reportS3Prefix",
        "prompt.$": "$.prompt",
        "style.$": "$$.Map.Item.Value"
      },
      "Iterator": {
        "StartAt": "QueueNarrativeAndWait",
        "States": {
          "QueueNarrativeAndWait": {
            "Type": "Task",
            "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
            "Parameters": {
              "QueueUrl": "${NarrativeQueueUrl}",
              "MessageBody": {
                "taskToken.$": "$$.Task.Token",
                "tenantId.$": "$.tenantId",
                "period.$": "$.period",
                "metricsKey.$": "$.metricsKey",
                "reportS3Prefix.$": "$.reportS3Prefix",
                "prompt.$": "$.prompt",
                "styleId.$": "$.style.id",
                "stylePrompt.$": "$.style.prompt"
              }
            },
            "TimeoutSeconds": 3600,
            "Retry": [
              {
                "ErrorEquals": ["States.Timeout", "States.TaskFailed"],
                "IntervalSeconds": 2,
                "BackoffRate": 2,
                "MaxAttempts": 3
              }
            ],
            "End": true
          }
        }
      },
      "ResultPath": "$.narratives",
      "Next": "MarkDone"
    },

    "MarkDone": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${MarkDoneLambdaArn}",
        "Payload": {
          "pk.$": "$.pk",
          "sk.$": "$.sk",
          "reportS3Prefix.$": "$.reportS3Prefix"
        }
      },
      "End": true
    }
  }
}

The above workflow essentially works in this way:

Athena has started.
The workflow pauses on a task token while Athena runs.
When Athena completes, an EventBridge-triggered Lambda calls SendTaskSuccess with the token.
For each style, the workflow sends an SQS message containing a task token and waits.
The worker does the OpenAI call, writes the narrative, and resumes the workflow using the task token.

A2) Athena resume handler

We need two Lambdas:

1) RegisterAthenaWait Lambda (called by Step Functions with .waitForTaskToken)

Stores taskToken keyed by queryExecutionId.

2) AthenaEventBridgeHandler Lambda (triggered by EventBridge)

Looks up the token by queryExecutionId and calls SendTaskSuccess/Failure.

The DynamoDB table to track the wait tokens looks like:

PK: ATHENA#<queryExecutionId>
Attributes: taskToken, pk, sk, createdAt

1) RegisterAthenaWait Lambda

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand } from "@aws-sdk/lib-dynamodb";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TOKENS_TABLE = process.env.TOKENS_TABLE!;

export const handler = async (event: any) => {
  const { taskToken, queryExecutionId, pk, sk } = event;

  await ddb.send(new PutCommand({
    TableName: TOKENS_TABLE,
    Item: {
      pk: `ATHENA#${queryExecutionId}`,
      sk: "WAIT",
      taskToken,
      jobPk: pk,
      jobSk: sk,
      createdAt: new Date().toISOString()
    }
  }));

  return {};
};

2) EventBridge handler (Athena completion → SendTaskSuccess)

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, GetCommand, DeleteCommand } from "@aws-sdk/lib-dynamodb";
import { SFNClient, SendTaskSuccessCommand, SendTaskFailureCommand } from "@aws-sdk/client-sfn";
import { AthenaClient, GetQueryResultsCommand } from "@aws-sdk/client-athena";

const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const sfn = new SFNClient({});
const athena = new AthenaClient({});

const TOKENS_TABLE = process.env.TOKENS_TABLE!;

export const handler = async (event: any) => {
  // Athena EventBridge event shape varies by detail-type; keep parsing defensive.
  const queryExecutionId = event?.detail?.queryExecutionId ?? event?.detail?.QueryExecutionId;
  const state = event?.detail?.currentState ?? event?.detail?.state ?? event?.detail?.CurrentState;

  if (!queryExecutionId) return;

  const tokenItem = await ddb.send(new GetCommand({
    TableName: TOKENS_TABLE,
    Key: { pk: `ATHENA#${queryExecutionId}`, sk: "WAIT" }
  }));
  if (!tokenItem.Item) return;

  const taskToken = (tokenItem.Item as any).taskToken as string;

  try {
    if (state === "SUCCEEDED") {
      // Fetch results or locate them in S3. For demo: fetch a small result set.
      const results = await athena.send(new GetQueryResultsCommand({
        QueryExecutionId: queryExecutionId,
        MaxResults: 1000
      }));

      const metrics = parseMetrics(results); // implement mapping to your metrics.json

      await sfn.send(new SendTaskSuccessCommand({
        taskToken,
        output: JSON.stringify({ metrics, queryExecutionId })
      }));
    } else {
      await sfn.send(new SendTaskFailureCommand({
        taskToken,
        error: "AthenaQueryFailed",
        cause: JSON.stringify({ queryExecutionId, state })
      }));
    }
  } finally {
    // Best-effort cleanup so we don't reuse tokens accidentally.
    await ddb.send(new DeleteCommand({
      TableName: TOKENS_TABLE,
      Key: { pk: `ATHENA#${queryExecutionId}`, sk: "WAIT" }
    }));
  }
};

function parseMetrics(_results: any) {
  ...
}

A3) SQS worker (Step Functions)

The worker must do this:
1) idempotency guard (SQS duplicates happen)
2) call OpenAI (using your subscription)
3) store narrative-${style}.json to S3
4) resume Step Functions using SendTaskSuccess/Failure

The DynamoDB table for the idempotency look like:

PK: NARRATIVE#<metricsKey>#<styleId>
SK: v1 (or a version string if you change format)
Attributes: s3Key, createdAt

Worker code

import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { DynamoDBDocumentClient, PutCommand, GetCommand } from "@aws-sdk/lib-dynamodb";
import { SFNClient, SendTaskSuccessCommand, SendTaskFailureCommand } from "@aws-sdk/client-sfn";
import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";

const s3 = new S3Client({});
const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const sfn = new SFNClient({});
const ssm = new SSMClient({});

const REPORT_BUCKET = process.env.REPORT_BUCKET!;
const NARR_TABLE = process.env.NARR_TABLE!;
const OPENAI_API_KEY_PARAM = process.env.OPENAI_API_KEY_PARAM!;

async function getOpenAiKey(): Promise<string> {
  const p = await ssm.send(new GetParameterCommand({
    Name: OPENAI_API_KEY_PARAM,
    WithDecryption: true
  }));
  const v = p.Parameter?.Value;
  if (!v) throw new Error("Missing OpenAI API key");
  return v;
}

export const handler = async (event: any) => {
  const key = await getOpenAiKey();

  for (const record of event.Records ?? []) {
    const msg = JSON.parse(record.body);

    const {
      taskToken,
      metricsKey,
      reportS3Prefix,
      prompt,
      styleId,
      stylePrompt
    } = msg;

    const narrPk = `NARRATIVE#${metricsKey}#${styleId}`;
    const narrSk = "v1";
    const narrativeKey = `${reportS3Prefix}narrative-${styleId}.json`;

    try {
      // 1) Idempotency guard (exactly-once effect)
      // Attempt to create the narrative record. If it exists, skip generation.
      try {
        await ddb.send(new PutCommand({
          TableName: NARR_TABLE,
          Item: { pk: narrPk, sk: narrSk, s3Key: narrativeKey, createdAt: new Date().toISOString() },
          ConditionExpression: "attribute_not_exists(pk) AND attribute_not_exists(sk)"
        }));
      } catch (e: any) {
        // Already exists: load and return success without re-running OpenAI
        const existing = await ddb.send(new GetCommand({
          TableName: NARR_TABLE,
          Key: { pk: narrPk, sk: narrSk }
        }));

        await sfn.send(new SendTaskSuccessCommand({
          taskToken,
          output: JSON.stringify({
            status: "ALREADY_DONE",
            narrativeKey: (existing.Item as any)?.s3Key
          })
        }));
        continue;
      }

      // 2) Load metrics.json (or pass metrics inline if small).
      // Here we assume it’s in S3: `${reportS3Prefix}metrics.json`
      // Read it and include it in the prompt.

      // 3) Call OpenAI (your subscription)
      const text = await callOpenAi({
        apiKey: key,
        userPrompt: prompt,
        stylePrompt
      });

      // 4) Store narrative artifact to S3
      await s3.send(new PutObjectCommand({
        Bucket: REPORT_BUCKET,
        Key: narrativeKey,
        ContentType: "application/json",
        Body: JSON.stringify({
          styleId,
          generatedAt: new Date().toISOString(),
          text
        })
      }));

      // 5) Resume Step Functions
      await sfn.send(new SendTaskSuccessCommand({
        taskToken,
        output: JSON.stringify({ status: "DONE", narrativeKey })
      }));
    } catch (err: any) {
      await sfn.send(new SendTaskFailureCommand({
        taskToken,
        error: "NarrativeFailed",
        cause: JSON.stringify({ message: err?.message ?? String(err), styleId, metricsKey })
      }));
    }
  }
};

async function callOpenAi(args: { apiKey: string; userPrompt: string; stylePrompt: string }): Promise<string> {
  const res = await fetch("https://api.openai.com/v1/responses", {
    method: "POST",
    headers: {
      "Authorization": `Bearer ${args.apiKey}`,
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      model: "gpt-5-mini",
      input: [
        { role: "system", content: "You write concise, accurate usage report narratives for SaaS metrics." },
        { role: "user", content: `${args.stylePrompt}\n\nUser request:\n${args.userPrompt}\n\nUse the metrics.json content as the source of truth.` }
      ]
    })
  });

  if (!res.ok) {
    const t = await res.text();
    throw new Error(`OpenAI error ${res.status}: ${t}`);
  }
  const json: any = await res.json();
  return json.output_text ?? JSON.stringify(json);
}

Notes

The idempotency guard is before calling OpenAI. That’s the whole point: prevent double spend when SQS redelivers.

Implementation B: Lambda Durable Functions

Docs

Durable functions overview: https://docs.aws.amazon.com/lambda/latest/dg/durable-functions.html
Durable execution SDK concepts (steps/waits/callbacks, retries): https://docs.aws.amazon.com/lambda/latest/dg/durable-execution-sdk.html
Callback completion APIs:
- SendDurableExecutionCallbackSuccess: https://docs.aws.amazon.com/lambda/latest/api/API_SendDurableExecutionCallbackSuccess.html
- SendDurableExecutionCallbackFailure: https://docs.aws.amazon.com/lambda/latest/api/API_SendDurableExecutionCallbackFailure.html

B1) Durable orchestrator

Key idea: for event-driven integration, we use callbacks:

create a callback id
send it to an external system (SQS worker/EventBridge handler)
resume execution when that external system completes the callback

Orchestrator

import { DurableContext } from "@aws/durable-execution-sdk-js";
import { SQSClient, SendMessageCommand } from "@aws-sdk/client-sqs";
import { AthenaClient, StartQueryExecutionCommand } from "@aws-sdk/client-athena";
import { S3Client, PutObjectCommand } from "@aws-sdk/client-s3";

const sqs = new SQSClient({});
const athena = new AthenaClient({});
const s3 = new S3Client({});

const NARRATIVE_QUEUE_URL = process.env.NARRATIVE_QUEUE_URL!;
const REPORT_BUCKET = process.env.REPORT_BUCKET!;
const TOKENS_TABLE = process.env.TOKENS_TABLE!; // callbackId mappings, implemented in submitters

type Style = { id: string; prompt: string };

type Input = {
  tenantId: string;
  period: string;
  metricsKey: string;
  metricsSpec: any;
  prompt: string;
  styles: Style[];
  reportS3Prefix: string;
};

export const handler = async (event: any, context: any) => {
  const dctx: DurableContext = (context as any).durable;

  const input = event as Input;

  // Step 1: start Athena query (checkpoint + retry)
  const queryExecutionId = await dctx.step(
    "start-athena",
    async () => {
      const out = await athena.send(new StartQueryExecutionCommand({
        QueryString: buildSql(input.metricsSpec),
        QueryExecutionContext: { Database: process.env.ATHENA_DB },
        ResultConfiguration: { OutputLocation: process.env.ATHENA_OUTPUT! }
      }));
      if (!out.QueryExecutionId) throw new Error("Missing QueryExecutionId");
      return out.QueryExecutionId;
    },
    {
      retry: {
        maxAttempts: 6,
        backoffRate: 2,
        initialDelayMs: 2000
      }
    }
  );

  // Step 2: wait for Athena completion via callback
  const athenaResult = await dctx.waitForCallback(
    "athena-done",
    async (callbackId) => {
      // Store callbackId keyed by queryExecutionId so EventBridge handler can complete it.
      // (Implementation shown in B2)
      await registerCallback("ATHENA", queryExecutionId, callbackId);
    },
    { timeout: { hours: 24 } }
  );

  // Step 3: store metrics.json once
  await dctx.step("write-metrics", async () => {
    await s3.send(new PutObjectCommand({
      Bucket: REPORT_BUCKET,
      Key: `${input.reportS3Prefix}metrics.json`,
      ContentType: "application/json",
      Body: JSON.stringify(athenaResult)
    }));
  });

  // Step 4: for each style → send SQS msg + wait for callback
  for (const style of input.styles) {
    await dctx.waitForCallback(
      `llm-${style.id}`,
      async (callbackId) => {
        // Enqueue work (one message per style)
        await sqs.send(new SendMessageCommand({
          QueueUrl: NARRATIVE_QUEUE_URL,
          MessageBody: JSON.stringify({
            callbackId,               // Durable resume token
            metricsKey: input.metricsKey,
            reportS3Prefix: input.reportS3Prefix,
            prompt: input.prompt,
            styleId: style.id,
            stylePrompt: style.prompt
          }),
          MessageAttributes: {
            styleId: { DataType: "String", StringValue: style.id },
            metricsKey: { DataType: "String", StringValue: input.metricsKey }
          }
        }));

        // Store mapping so the worker can be idempotent and handlers can correlate if needed.
        await registerCallback("LLM", `${input.metricsKey}#${style.id}`, callbackId);
      },
      { timeout: { hours: 1 } }
    );
  }

  // Step 5: done
  return {
    status: "DONE",
    reportS3Prefix: input.reportS3Prefix
  };
};

function buildSql(_spec: any): string {
  // Deterministic SQL builder from metricsSpec.
  return "SELECT ....";
}

async function registerCallback(kind: "ATHENA" | "LLM", key: string, callbackId: string) {
  // Intentionally abbreviated for article brevity: use the same DynamoDB PutItem pattern as the Step Functions token table.
  ...
}

B2) Athena resume handler (Durable)

The Athena resume handler is intentionally almost identical to the Step Functions version; the only difference is the API used to resume execution.

Instead of calling SendTaskSuccess / SendTaskFailure, the handler calls:

SendDurableExecutionCallbackSuccess
SendDurableExecutionCallbackFailure

The event source, correlation logic, and failure semantics remain unchanged.

B3) SQS worker (Durable)

The SQS worker logic is identical to the Step Functions version.

The only difference is how the worker resumes the paused orchestration:

Step Functions: SendTaskSuccess / SendTaskFailure
Durable Functions: SendDurableExecutionCallbackSuccess / SendDurableExecutionCallbackFailure

All other concerns—idempotency, retries, OpenAI rate limits, and S3 materialisation—remain unchanged.

Cost comparison

Below is a cost comparison that isolates the differences between the two orchestration options. Everything else (Athena scan costs, S3 storage, DynamoDB read/write, OpenAI tokens, etc.) is essentially the same for both designs.

Step Functions Standard: $0.000025 per state transition (=$0.025 per 1,000 transitions)

Pricing page: https://aws.amazon.com/step-functions/pricing/
Lambda Durable Functions: durable operations are billed separately. The Lambda pricing page’s example uses $8.00 per 1M durable operations, plus data written/retained charges.

https://aws.amazon.com/lambda/pricing/ (see “Lambda Durable Functions Pricing” section)
SQS Standard: ~$0.40 per 1M API requests (Send/Receive/Delete)

https://aws.amazon.com/sqs/pricing/

Let:

S = number of narrative styles requested in one report request (e.g., S=3: executive/technical/anomalies)

For Step Functions Standard: we have roughly:

StartAthena → 1 transition
RegisterAndWaitForAthena → 1 transition
PutMetricsToS3 → 1 transition
NarrativesMap overhead → 1 transition
Inside Map: per style, QueueNarrativeAndWait → ~1 transition each (plus Map bookkeeping)
MarkDone → 1 transition

We could say:

Transitions ≈ 5 + S

For S = 3:

transitions ≈ 8
cost per report ≈ 8 * 0.000025 = $0.000200

Volumes:

1,000 reports → $0.20
10,000 reports → $2.00
100,000 reports → $20.00

For the Lambda Durable Functions: we have:

Start Athena: 1 step
Wait for Athena callback: 1 callback wait
Write metrics: 1 step
For each style: 1 callback wait (the submitter + wait)
(Plus orchestration lifecycle operations)

We could say:

Durable operations ≈ 4 + S

For S = 3:

durable ops ≈ 10
ops cost per report ≈ 10 * (8 / 1,000,000) = $0.000080

Volumes:

1,000 reports → $0.08
10,000 reports → $0.80
100,000 reports → $8.00

But durable also adds:

Data written (payload checkpoints)
Data retained (GB-month) if you keep a durable history

These depend heavily on:

checkpoint payload sizes
how long callbacks wait (hours vs days)
retention window

What is actually different?

Step Functions
- explicit workflow definition (ASL)
- built-in execution history & visual debugging
- managed service integrations, clear retry policies at the workflow level
- task token callback pattern for SQS
Durable Functions
- code-first orchestration
- durable checkpoints + deterministic replay
- callback IDs completed via Lambda APIs (SendDurableExecutionCallbackSuccess/Failure)

Conclusion

This is not about Step Functions and Durable Functions competing against each other. They are simply two ways to build the same thing, and the hard problems do not disappear by changing the orchestration tool.

Lambda Durable Functions are not revolutionary (my view). What actually differs between the two approaches is where complexity lives.

Step Functions externalise orchestration into a managed service.
Durable Functions move the workflow into the codebase.

From a cost perspective, the orchestration cost tends to be in a similar range for both approaches.

If a team has little or no experience with ASL and prefers to keep orchestration logic close to application code, Lambda Durable Functions can be a very good fit.

When (Not If) Containers Misbehave

Daniele Frasca — Wed, 17 Sep 2025 06:27:30 +0000

In my previous article, Fargate + Lambda are better together, I introduced the concept of a hybrid architecture where I can route traffic between ECS Fargate and Lambda based on real-time conditions.

The most common argument I know about is:

Why not just use containers for everything?

In this article, I try to explain why the "containers are cheaper" is an oversimplification.

The Hidden Cost Equation

Before I go into details, let's address the costs (hopefully I did the calculation correctly):

Assuming typical production capacity:

Lambda execution time: ~118ms average (1GB memory allocation)
Fargate task size: 2vCPU/4GB running continuously
Daily capacity: ~2M requests/day per task
Monthly capacity: ~60M requests/month per task

Note: These cost calculations are based on Lambda functions with 118ms execution time. The actual costs will vary significantly based on application's performance characteristics.

Monthly Cost Comparison:

Monthly Requests	Tasks Needed	Lambda Cost	Fargate Cost	Winner
1M	1	$2,17	$56,92	Lambda 96% cheaper
10M	1	$21,67	$56,92	Lambda 62% cheaper
25M	1	$54,17	$56,92	Lambda 5% cheaper
50M	1	$108,33	$56,92	Fargate 47% cheaper
65M	2	$140,83	$113,84	Fargate 19% cheaper
100M	2	$216,67	$113,84	Fargate 47% cheaper
130M	3	$281,67	$170,76	Fargate 39% cheaper

Daily Cost Comparison:

Daily Requests	Tasks Needed	Lambda Cost	Fargate Cost	Winner
1M	1	$2,17	$1,90	Fargate 12% cheaper
10M	5	$21,67	$9,50	Fargate 56% cheaper
25M	12	$54,17	$22,80	Fargate 58% cheaper
50M	24	$108,33	$45,60	Fargate 58% cheaper
65M	31	$140,83	$58,90	Fargate 58% cheaper
100M	47	$216,67	$89,30	Fargate 59% cheaper
130M	61	$281,67	$115,90	Fargate 59% cheaper

The tables show that containers become cheaper around 50M requests/month or over 1M requests daily.

If I consider the human cost, based on European senior developer rates (Stack Overflow 2024 Survey - €75K average = €36/hour):

Engineering Effort	Time Investment	Cost
Load testing and optimization	2 weeks (80 hours)	€2.880
Traffic controller development	2 weeks (80 hours)	€2.880
Production fixes and fine-tuning	1 week (40 hours)	€1.440

Total Fargate human cost: €7.200

With Lambda, I skip most of this complexity:

No failure scenario testing: AWS handles all for me
No traffic controller: No need to invent it
No production fine-tuning: It just works out of the box

With 14 operational scenarios, I need to simulate and prepare for each failure mode, while Lambda eliminates all of them through abstraction.

Even when Fargate appears cheaper on infrastructure costs (around 50M+ requests/month or 1M+ requests/day). The €7.200 engineering investment only becomes cost-effective after many months.

Payback Calculation Formula:

Monthly Savings = (Lambda Daily Cost - Fargate Daily Cost) × 30 days
Payback Period (months) = €7,200 ÷ Monthly Savings (USD)

Examples:
• 10M requests/day: ($21.67 - $9.50) × 30 = $365.10/month
  Payback = €7,200 ÷ $365.10 = 19.7 months

• 25M requests/day: ($54.17 - $22.80) × 30 = $941.10/month  
  Payback = €7,200 ÷ $941.10 = 7.6 months

Human Cost Payback:

Traffic Volume	Daily Savings	Monthly Savings	Payback Period
1M requests/month	N/A	-$54.75	Never (Lambda cheaper)
10M requests/day	$12.17	$365.10	19.7 months
25M requests/day	$31.37	$941.10	7.6 months
50M requests/day	$62.73	$1,881.90	3.8 months

Based on the table, the €7.200 human cost works are only cost-effective from 7-8 months at extremely high traffic (25M+ requests/day) to multiple years at moderate traffic levels, assuming:

No major architectural changes
No scaling issues requiring rework
No additional failure modes discovered

What Makes Fargate "Complex"

Let's see what it takes to run containers in production:

  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub "${AWS::StackName}-cluster"
      CapacityProviders:
        - FARGATE
        - FARGATE_SPOT
      DefaultCapacityProviderStrategy:
        - CapacityProvider: FARGATE
          Weight: 2  # 40% on-demand for stability
        - CapacityProvider: FARGATE_SPOT
          Weight: 3  # 60% spot for cost savings
      ClusterSettings:
        - Name: containerInsights
          Value: enhanced

Without the Amazon ECS Container Insights metrics enabled, I would not be able to monitor metrics that could cause downtime, like:

CPU
Memory
RunningTaskCount

Most teams start with what I call the 'monolithic service trap' - one big ECS service that seems simple until it isn't.

  MyService:
    Type: AWS::ECS::Service
    Properties:
      ServiceName: !Sub "${AWS::StackName}-my-service"
      Cluster: !Ref ECSCluster
      TaskDefinition: !Ref MyTaskDefinition
      LaunchType: FARGATE
      PlatformVersion: LATEST
      DeploymentConfiguration:
        Strategy: ROLLING
        MaximumPercent: 200          # Allow 100% extra capacity during deployment  
        MinimumHealthyPercent: 100   # Keep all current tasks running until new ones are healthy
        DeploymentCircuitBreaker:
          Enable: true
          Rollback: true
      DeploymentController:
        Type: ECS
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: DISABLED
          SecurityGroups:
            - !Ref ECSSecurityGroupId
          Subnets: !FindInMap [!Ref StageName, !Ref "AWS::Region", PrivateSubnets]
      LoadBalancers:
        - ContainerName: my-container
          ContainerPort: 3000
          TargetGroupArn:
            Fn::ImportValue:
              !Sub "${ALBStackName}-My-ECS-TargetGroup-Arn"
      HealthCheckGracePeriodSeconds: 60

  MyTaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub "${AWS::StackName}-my-task-definition"
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      Cpu: 2048
      Memory: 4096
      ExecutionRoleArn: !GetAtt ECSExecutionRole.Arn
      TaskRoleArn: !GetAtt ECSTaskRole.Arn
      ContainerDefinitions:
        - Name: my-container
          Image: !Ref MyImageUri
          Essential: true
          PortMappings:
            - ContainerPort: 3000
              Protocol: tcp
          Environment:
            - Name: AWS_REGION
              Value: !Ref "AWS::Region"
            - Name: LOG_LEVEL
              Value: !FindInMap [LogLevel, !Ref StageName, level]
            - Name: NODE_OPTIONS
              Value: "--max-old-space-size=3072"  # Use 75% of 4GB memory for heap
      ....

The problems with this approach are:

Single Point of Failure: One bad deployment kills everything
All-or-Nothing Updates: Can't update just something without affecting something else
Blast Radius: one endpoint path can bring all down

I like to organise my architecture based on specific parameters, and they can vary based on the application. I am working on an international streaming application, so I could potentially split my services into:

country (maybe it is better to have ECS per country)
type of users
type of subscriptions
type of platform

The above are examples, but they have some obvious advantages:

Fault Isolation
Independent Deployments
Smaller Blast Radius
Resource Isolation
Independent Capacity
Observability
Scalability

A single monolithic service creates availability and scalability bottlenecks, where the failure of one component can impact the entire system.
On the opposite extreme, there is excessive fragmentation. When services are correctly isolated, a bug or outage in one service cannot directly impact other services. This isolation is particularly valuable when services have different reliability requirements or face varying load patterns, as failures remain localised instead of bringing all down.

A lesson I have learned is that service decomposition becomes economically justified when different components have significantly different resource traffic patterns. Each component scales independently according to its actual needs. Without this separation, the entire system must be provisioned for peak load across all components, often resulting in extra overprovisioning and wasted resources.

I want to highlight some important configurations for AWS::ECS::Service and AWS::ECS::TaskDefinition

Deployment Strategy

While AWS ECS supports blue/green deployments through CodeDeploy integration, this feature does not work when ALB target groups dynamically route traffic between ECS and Lambda functions.
Blue/green deployments require dedicated target groups for each deployment stage, which conflicts with my setup.
The rolling deployment strategy provides zero-downtime deployments while maintaining compatibility with my hybrid traffic management system.

Resource Allocation: CPU and Memory Sizing

Cpu: 2048    # 2 vCPU - Increased from 1024 due to load testing findings
Memory: 4096 # 4GB - Increased from 2048 due to memory pressure and GC issues

The resource allocation reflects the results from the load testing. Initially, I had (1 vCPU, 2GB RAM), but it resulted in cascading performance issues during traffic spikes:

CPU Saturation
Memory Pressure

Fargate Cold Start

As Vlad Ionescu pointed out in his post, Fargate takes its time to scale up. From what I can see, it takes up to 5 minutes. Always, thanks to the load testing, I found out that newly launched tasks were receiving full traffic upon ALB target registration, before completing their initialisation routines, because they were considered 'healthy', which contributed to 502/503 errors.

To mitigate this, I added an extra configuration:

  MyServiceSTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      TargetGroupAttributes:
        - Key: slow_start.duration_seconds
          Value: "60"

In the end, by increasing resource allocation to 2 vCPU/4GB and implementing a 60-second ALB slow start period, I achieved:

Reduced Error Rates: The combination eliminated the 502/503 error spikes during scaling events
Better Cost Efficiency: While individual tasks cost more, we need fewer of them and experience fewer scaling thrashes

The 14 potential pitfalls

Lambda is a full delegation to AWS. It is magic.

ECS Fargate has its complexity, but this is the price of control.

Here are all the cases I found while testing

Failure Category	What Can Go Wrong	Real-World Impact
🔥 Emergency Failures	Zero running tasks available	🚨 Complete service outage - immediate Lambda failover required
⚠️ Capacity Issues	Down to minimum viable tasks	📉 Gradual failure prevention - start shifting to Lambda
💀 Critical Capacity	Only one task left running	💣 Last chance before outage - emergency traffic routing
🚀 Traffic Overload	Requests exceed Lambda capacity	⬆️ Lambda can't handle load - need ECS scaling
💥 Traffic Spike	Sudden burst traffic overwhelms system	⚡ Viral content or DDoS - hybrid ECS+Lambda mode
🌊 Sustained Load	High traffic persists for extended period	💰 Cost optimization - shift to ECS-heavy hybrid mode
😴 Low Traffic	Traffic drops below cost-effective threshold	💸 ECS waste - switch back to Lambda
💻 CPU Exhaustion	CPU utilization reaches warning threshold	🥵 Resource competition - enable Lambda overflow
🔴 CPU Emergency	CPU utilization hits critical levels	🚒 Cascade failure risk - emergency Lambda routing
💾 Memory Pressure	Memory usage approaches warning limits	🤯 Memory leaks detected - Lambda overflow
‼️ Memory Emergency	Memory usage hits critical threshold	🆘 OOM kills imminent - immediate failover
⬇️ Scale-In Problems	Auto-scaling fails to reduce capacity efficiently	💵 Idle tasks burning money - gradual scale-down
💀 Spot Interruptions	AWS reclaims spot instances	🌩️ Multiple tasks lost suddenly
🎯 Path-Specific Issues	Individual service failures	🚧 Blast radius contained to one service

Notice how many scenarios demand "Lambda failover" or "Lambda overflow" as solutions. This is because Lambda has a unique strength: it can instantiate thousands of instances in milliseconds.

This remarkable ability comes at a cost, as I pay more because AWS manages everything for me. But when Fargate faces any of these 14 challenges, Lambda's instant scalability becomes priceless.

Zero capacity planning becomes an asset when you need emergency capacity instantly
Automatic scaling becomes crucial when Fargate scaling can't keep up with traffic spikes
No infrastructure management means no infrastructure bottlenecks during a crisis
Isolation per request ensures consistent performance regardless of load

Performance trade-offs

Fargate Performance Degrades Under Load: Fargate latency increases as more resources are consumed within the same task. A task handling 1 request performs very differently from one handling thousands simultaneously
Lambda Maintains Consistent Performance: Lambda stays consistent because it's 1 request per execution environment
Different Performance Profiles: Fargate excels at sustained moderate load (for example, p95: ~30ms ecs vs ~70ms lambda), while Lambda excels at consistent performance regardless of system load
Cost vs Performance Trade-off: You pay Lambda's higher price for guaranteed isolation and instant scalability
Failure Recovery: Lambda's instant provisioning makes it ideal for emergency scenarios when Fargate fails

Fargate performs better in stable conditions but degrades under stress, and its total cost of ownership is so high that its complexity pays off only in the long term.
Lambda performs consistently but at a higher cost per request, but this higher pricing compensates for operational simplicity and faster delivery cycles.

It's Not About Cost, It's About Risk

After running this hybrid architecture in production, here's what I've learned:

Lambda is not more expensive
Containers are not cheaper
Hybrid architectures give you the best of both worlds

What's Next: Traffic Controller

I'll share the Traffic Controller implementation that makes this hybrid magic possible. The Traffic Controller is a component that dynamically routes traffic between Lambda and ECS based on real-time traffic patterns.
A component that monitors CloudWatch alarms, understands task-aware load distribution, and makes routing decisions.

Fargate + Lambda are better together

Daniele Frasca — Wed, 03 Sep 2025 04:00:00 +0000

After many years working with Serverless at a certain scale, I am starting to wonder about some things. I have been fortunate enough to attend numerous conferences where I have learned about the great potential of serverless computing and the various options available. At the same time, I noticed a separation between ECS Fargate and Lambda that makes it difficult for many to make a choice.

If I ask around, I hear the same story:

Fargate is cheaper at scale
Lambda is better for spiky traffic

I often wonder why running both of them simultaneously with the same code base is not possible.
How difficult can it be? Underground rumours, now three years old, suggested that AWS is working on this. However, at each re:Invent, there has been little to nothing announced for the next generation of Serverless applications, so I decided to test it myself.

The AWS option:

https://github.com/awslabs/aws-lambda-web-adapter

Features:

Run web applications on AWS Lambda
Supports Amazon API Gateway Rest API and HTTP API endpoints, Lambda Function URLs, and Application Load Balancer
Supports Lambda managed runtimes, custom runtimes and Docker OCI images
Supports any web frameworks and languages, with no new code dependency to include
Automatic encode binary response
Enables graceful shutdown
Supports response payload compression
Supports response streaming
Supports non-HTTP event triggers

With this, you now have 1 code that runs with both services.

If I do not want to use it for any reason, it is also easier to go around.

I still have my regular Lambda handler:

// init outside of the handler

export const handler = async (event: ALBEvent): Promise<ALBResult> => {
   // code
}

Now you need your ECS entry point:

import { ALBEvent, ALBResult } from "aws-lambda/trigger/alb";
import express, { Request, Response } from "express";

// Import your existing handlers
import { handler as myLambdaHandler } from "./handlers/myLambda/index";

const app = express();
const port = parseInt(process.env.PORT || "3000", 10);

// Middleware
app.use(express.json());

app.get("/yourPath", async (req: Request, res: Response) => {
  try {
    const albEvent = createALBEvent(req);
    const result = await myLambdaHandler(albEvent);
    sendALBResponse(res, result);
  } catch (error) {
    console.log(
      JSON.stringify({
        level: "error",
        message: "Error in myLambdaHandler",
        error: error?.toString(),
        stack: error?.stack,
      }),
    );
    res.status(500).json({ error: "Internal server error" });
  }
});

// Helper function to convert Express req/res to ALB event format
function createALBEvent(req: Request): ALBEvent {
  return {
    requestContext: {
      elb: {
        targetGroupArn: "arn:aws:elasticloadbalancing:region:account:targetgroup/target-group-name",
      },
    },
    httpMethod: req.method,
    path: req.path,
    queryStringParameters: (req.query as { [key: string]: string }) || null,
    headers: (req.headers as { [key: string]: string }) || {},
    body: req.method === "POST" || req.method === "PUT" ? JSON.stringify(req.body) : null,
    isBase64Encoded: false,
  };
}

// Helper function to send ALB response through Express
function sendALBResponse(res: Response, albResult: ALBResult): void {
  res.status(albResult.statusCode);

  if (albResult.headers) {
    Object.entries(albResult.headers).forEach(([key, value]) => {
      if (typeof value === "string") {
        res.set(key, value);
      } else if (typeof value === "number") {
        res.set(key, value.toString());
      }
    });
  }

  if (albResult.body) {
    if (albResult.isBase64Encoded) {
      res.send(Buffer.from(albResult.body, "base64"));
    } else {
      // Handle both string and parsed JSON content
      try {
        // Try to parse as JSON first
        const parsed = JSON.parse(albResult.body);
        res.json(parsed);
      } catch {
        // If not JSON, send as plain text
        res.send(albResult.body);
      }
    }
  } else {
    res.end();
  }
}


// Start the server
app.listen(port, () => {
  console.log(
    JSON.stringify({
      level: "info",
      message: "Server running",
      port: port,
      runtime: process.env.AWS_LAMBDA_FUNCTION_NAME ? "lambda": "ecs",
      region: process.env.REGION || "unknown",
      authType: process.env.AUTH_TYPE || "unknown",
    }),
  );
});

The wrapper above will be referenced from the Dockerfile used for ECS.

NOTE

The code above is the classic leaky abstraction problem. The ECS web API shouldn't need to know about Lambda's ALB event format. This creates tight coupling between the business logic and AWS Lambda's specific interface. The leaky abstraction can be fixed with hexagonal architecture principles, where your business logic is at the centre, protected from external framework changes

The cool part?

Imagine two applications that are complementary to each other:

AWS Fargate provides a steady, always‑ready core for predictable, high‑volume or latency‑sensitive requests,
AWS Lambda acts as an elastic reflex that expands instantly for sudden bursts or temporary overload.

Using Lambda only often means paying a higher per-request price during long, sustained traffic and occasionally experiencing cold start delays. Using Fargate only forces provisioning for peak usage (wasting money off-peak) and risks overloads and outages while autoscaling catches up.

Combining them removes that trade‑off because I have a unified codebase that can run in both environments, traffic can be smoothly balanced (for example 30/70 → 60/40 → 85/15) based on real demand, and protective "task‑aware" routing ensures a small number of container tasks is never flooded while Lambda stands by as a safety for any spike and or ECS outage when they occurs.

This hybrid setup enables many things, like:

Move one endpoint or path at a time
Safer deployments (shift traffic away before replacing tasks), and more precise cost control
Resilience in terms of services, what are the chances that both of them will stop working
Faster innovation, you can redirect part of the traffic to one service before making it stable

Is Fargate really cheaper than Lambda?

On paper yes, but in reality, no.
Containers (Fargate in this case) appear more cost-effective because we have a steady vCPU/GB-hour rate compared to Lambda.
The reality check and the data that I have show the opposite. I have to choose task sizes (vCPU/memory ratios), tune autoscaling targets, decide on scale-in cooldowns, set alarm thresholds, pick concurrency buffers, warm new tasks before they receive load, and revisit all of this each time traffic shape, code efficiency, or dependency latency changes.
Each change is another way to under-provision (causing throttling, retries, and user latency) or over-provision (resulting in silent waste, which is the default solution in 99% of cases).

Every tech decision has an associated labour cost, including observability dashboards, runbooks, on-call pages, postmortems, tuning cycles, regression risks after refactors, and load testing. By contrast, Lambda pays the raw unit price, which I refer to as the enterprise fee; however, the platform absorbs many failure modes that I would otherwise have to plan for and work around.

What's Next: When (Not If) Containers Misbehave

This article is the high‑level "why". In the next part, I will try to go deeper into the uncomfortable truth behind the simple phrase "just run it on container" and touch on the scenarios where container workloads actually fail in production and then I will show you how to map each of those failure modes to a traffic controller I built that proactively adapt traffic between Lambda and ECS based on real-time traffic patterns.

Adaptive Compression for Cache Systems with Hexagonal Architecture

Daniele Frasca — Wed, 16 Apr 2025 04:00:00 +0000

Any API at some significant scale adopts some form of caching for optimisation by reducing database load and speeding up response times. However, as applications scale, the size and variety of cached data can have some challenges:

Large objects consume excessive memory in the cache
Network bandwidth becomes a bottleneck for data transfer
Cache costs increase with data size

As there is no one solution to cover all, I had an idea:

Design a caching system that could efficiently handle diverse data sizes and types
Create a compression strategy that adapts to different data characteristics automatically
Ensure the solution works across different caching services (Redis, Memcached...)
Guarantee that compression benefits outweigh its processing costs

With hexagonal architecture, I can decouple the caching logic from specific providers:

// Sample architecture overview
interface CachePort {
  get(key: string): Promise<any>;
  set(key: string, value: any, ttl?: number): Promise<boolean>;
  getBatch(keys: string[]): Promise<any[]>;
  setBatch(items: {key: string, value: any}[], ttl?: number): Promise<boolean[]>;
}

class RedisAdapter implements CachePort { /* ... */ }
class MemcachedAdapter implements CachePort { /* ... */ }
class OtherAdapter implements CachePort { /* ... */ }

This architecture enabled compression implementation at the port level, ensuring it was available regardless of the cache provider used.

The adaptable compression module is based on the following:

Data size (small objects aren't compressed)
Operation type (single vs batch operations)
Content type (some data compresses better than others)

The compression implementation looks like this:

import * as zlib from "zlib";

export enum CompressionType {
  BROTLI = "br",
  GZIP = "gzip",
  NONE = "none",
}

// Compression markers (first byte of the compressed data)
const COMPRESSION_MARKER = {
  NONE: 0,
  BROTLI: 1,
  GZIP: 2,
};

export const compressionStats = {
  enabled: process.env.ENV !== "prod", // Disable in production by default
  totalBytesInput: 0,
  totalBytesOutput: 0,
  totalSaved: 0,
  compressionRatio: 0,
  itemsCompressed: 0,
  reset: function () {
    this.totalBytesInput = 0;
    this.totalBytesOutput = 0;
    this.totalSaved = 0;
    this.itemsCompressed = 0;
    this.compressionRatio = 0;
  },
};

export function compress(value: string, options?: { preferredCompression?: CompressionType }): Uint8Array {
  const preferredCompression = options?.preferredCompression || CompressionType.GZIP;
  let compressedData: Uint8Array;
  let marker: number;
  const originalSize = value.length;

  switch (preferredCompression) {
    case CompressionType.BROTLI:
      console.debug(`Brotli compression: ${originalSize} bytes input`);
      compressedData = zlib.brotliCompressSync(Buffer.from(value));
      marker = COMPRESSION_MARKER.BROTLI;
      break;

    case CompressionType.GZIP:
      console.debug(`GZIP compression: ${originalSize} bytes input`);
      compressedData = zlib.gzipSync(Buffer.from(value));
      marker = COMPRESSION_MARKER.GZIP;
      break;

    default:
      // NONE type - no actual compression, add format marker
      console.debug(`No compression: ${originalSize} bytes`);
      compressedData = Buffer.from(value);
      marker = COMPRESSION_MARKER.NONE;
  }

  // Create a new buffer with marker byte at the beginning
  const result = new Uint8Array(compressedData.length + 1);
  result[0] = marker;
  result.set(compressedData, 1);

  // Update stats only if enabled (conditional)
  if (compressionStats.enabled) {
    compressionStats.totalBytesInput += originalSize;
    compressionStats.totalBytesOutput += result.length;
    compressionStats.totalSaved += originalSize - result.length;
    compressionStats.itemsCompressed += 1;
    compressionStats.compressionRatio = compressionStats.totalBytesOutput / compressionStats.totalBytesInput;

    // Periodic reset to avoid potential memory issues in long-running processes
    if (compressionStats.itemsCompressed > 1000000) {
      // Reset after 1 million items
      compressionStats.reset();
    }
  }

  console.debug(
    `Compression: ${originalSize} → ${result.length} bytes (${Math.round((result.length / originalSize) * 100)}%)`,
  );

  return result;
}

export function decompress(data: Uint8Array | string): string {
  if (typeof data === "string") {
    return data; // If it's a string, assume it's uncompressed
  }

  if (data.length === 0) {
    return ""; // Empty data
  }

  const marker = data[0];
  const compressedData = data.subarray(1); // Remove the marker byte

  try {
    let result: string;

    switch (marker) {
      case COMPRESSION_MARKER.BROTLI:
        result = zlib.brotliDecompressSync(compressedData).toString();
        break;

      case COMPRESSION_MARKER.GZIP:
        result = zlib.gunzipSync(compressedData).toString();
        break;

      case COMPRESSION_MARKER.NONE:
        result = Buffer.from(compressedData).toString();
        break;

      default:
        // For backwards compatibility and resiliency
        return Buffer.from(data).toString();
    }

    return result;
  } catch (error) {
    console.error(`Decompression error with marker ${marker}: ${error}`);
    // Last resort fallback
    return Buffer.from(compressedData).toString();
  }
}

The key aspect of this compression module is making the data "self-describing." By embedding a single-byte marker at the beginning of each compressed payload, the data contains information about how it was compressed, allowing me to determine which decompression algorithm to use. Uncompressed data can flow through the same pipeline as compressed data, and I can interchange my preferred compression at any moment. Another nice feature of this module is that I can add a new compression algorithm anytime. If, in the future, Node supports, for example, Zstd, I will add a new marker value. The marker-based compression system represents a simple and extendable pattern that brings significant power to data compression.

The cache client uses the compression module with adaptive logic, and it can look like this:

class CacheClient {
  private cacheAdapter: CachePort;

constructor(adapter: CachePort, options: Options)  {
    this.cacheAdapter = adapter;
    this.compressionThreshold = options.compressionThreshold || 2048; // 2KB default
    this.preferredCompression = options.preferredCompression || "GZIP";
    this.batchSize = options.batchSize || 100;
  }
// Single item storage with adaptive compression
  async set(key, value, ttl) {
    const stringValue = JSON.stringify(value);
    let compressedValue;

    // Apply adaptive compression logic based on data size
    if (stringValue.length < this.compressionThreshold) {
      // Small values skip compression - not worth the CPU cost
      compressedValue = value;
    } else if (stringValue.length > 50 * 1024) {
      // Large values (>50KB) use the configured preferred encoding
      compressedValue = {
        compressed: true,
        data: compress(stringValue, { encoding: this.preferredCompression })
      };
    } else {
      // Medium values always use GZIP for better speed/ratio balance
      compressedValue = {
        compressed: true,
        data: compress(stringValue, { encoding: "GZIP" })
      };
    }

    return this.cacheAdapter.set(key, compressedValue, ttl);
  }

  // Retrieval with automatic decompression
  async get(key) {
    const result = await this.cacheAdapter.get(key);

    if (!result) return null;

    // Handle compressed data
    if (result.compressed && result.data) {
      const decompressedValue = decompress(result.data);
      return JSON.parse(decompressedValue);
    }

    return result;
  }

  // Batch storage with adaptive logic
  async setBatch(items, ttl) {
    const batchSize = items.length;
    const processedItems = [];

    for (const item of items) {
      const stringValue = JSON.stringify(item.value);

      // Adjust threshold based on batch size
      // As batch size increases, compression threshold decreases
      const effectiveThreshold = Math.max(1024, this.compressionThreshold / Math.sqrt(batchSize));

      // Determine best compression algorithm for this batch item
      let compressionType;
      if (stringValue.length < effectiveThreshold) {
        // Skip compression for very small values
        processedItems.push({ key: item.key, value: item.value });
        continue;
      } else if (batchSize >= 20 || stringValue.length > 10 * 1024) {
        // Use Brotli for large batches or large individual items
        compressionType = "BROTLI";
      } else {
        // Use GZIP for smaller batches with medium-sized items
        compressionType = "GZIP";
      }

      processedItems.push({
        key: item.key,
        value: {
          compressed: true,
          data: compress(stringValue, { encoding: compressionType })
        }
      });
    }

    // Process in chunks to avoid overwhelming the cache service
    const results = [];
    for (let i = 0; i < processedItems.length; i += this.batchSize) {
      const chunk = processedItems.slice(i, i + this.batchSize);
      const chunkResults = await this.cacheAdapter.setBatch(chunk, ttl);
      results.push(...chunkResults);
    }

    return results;
  }

  // Batch retrieval with automatic decompression
  async getBatch(keys) {\
    const results = [];

    for (let i = 0; i < keys.length; i += this.batchSize) {
      const chunk = keys.slice(i, i + this.batchSize);
      const chunkResults = await this.cacheAdapter.getBatch(chunk);

      for (const result of chunkResults) {
        if (!result) {
          results.push(null);
        } else if (result.compressed && result.data) {
          // Decompress and parse
          const decompressedValue = decompress(result.data);
          results.push(JSON.parse(decompressedValue));
        } else {
          results.push(result);
        }
      }
    }

    return results;
  }
}

The setBatch operation adjusts the compression threshold based on batch size. As the batch size increases, I lower the size threshold at which compression activates. With a default threshold of 2KB:

Single operation: 2048 bytes
Batch of 4 items: ~1024 bytes
Batch of 25 items: ~409 bytes

It determines which compression algorithm to use, such as Brotli for large batches or large individual items and GZIP for smaller batches with medium-sized items. Additionally, chunking is utilised to prevent overwhelming the cache service.

I have run multiple tests:

Tests both compression types (GZIP, Brotli) and uncompressed operations
Measures performance metrics across different operations and data sizes

I have noticed the following:

Compression ratios: Both GZIP and Brotli achieve excellent compression (≈70-99%)
Single operations: GZIP is faster for SET operations (≈100ms vs 217ms for Brotli)
Batch operations: Brotli provides better read performance (≈18% faster than GZIP)
Memory efficiency: Compressed storage uses only ≈27-31% of the original size

If we are talking about the size, my conclusion is:

Small values (<2KB): Not compressed, minimal overhead applied
Medium values (2KB-50KB): Compressed with GZIP for better speed
Large values (>50KB): Compressed with configured algorithm
Small batches (<20 items): Use GZIP for better speed
Large batches (≥20 items): Use Brotli for better compression

The compression ratio is the following:

Data Type	Original Size	GZIP Size	Brotli Size	Compression Ratio
Medium JSON	4KB	1.2KB	1.1KB	73-75%
Large batch	5KB per item	55 bytes	29 bytes	99.4-99.5%
String data	2KB	46 bytes	27 bytes	98-99%
Large object	100KB	149 bytes	-	99.9%

Compression decisions are made at write time only (set or setBatch). This means:

If I set a 1KB item individually (uncompressed), then later include it in a getBatch operation, it remains uncompressed.
If I setBatch 25 items of 1KB each (compressed due to batch size), then later get a single item, it will be automatically decompressed.
The client automatically handles decompression regardless of which operation was used to retrieve the data.

Some scenarios while running tests:

Scenario	What Happens	Algorithm	Reason	Notes
10 items × 5KB each	All compressed individually	GZIP	Each item > 2KB, batch size < 20	Each 5KB item → 55 bytes (99% reduction)
25 items × 5KB each	All compressed individually	Brotli	Each item > 2KB, batch size ≥ 20	Each 5KB item → 29 bytes (99.4% reduction)
5 items × 15KB each	All compressed individually	Brotli	Individual items > 10KB	Better compression for large values
25 items × 1KB each	All compressed individually	Brotli	Threshold adjusted for large batches	Items below standard threshold but compressed due to batch size
10 items × 1KB each	May be compressed	GZIP	Effective threshold becomes ~1024 bytes	Compression depends on exact size vs threshold
25 items × 1KB each	All compressed	Brotli	Large batch lowers threshold to ~1024 bytes	Demonstrates adaptive threshold adjustment
10 items × 2KB each	All compressed	GZIP	At standard threshold, batch size < 20	Standard compression behavior
25 items × 2KB each	All compressed	Brotli	Above threshold, batch size ≥ 20	Algorithm selection based on batch size

To make it short in terms of performance, I have found the following:

GZIP is consistently faster for medium and large data sizes • GZIP is approximately 37.9% faster than Brotli for larger data
Decompression speeds are comparable between GZIP and Brotli (< 5ms difference)

Data Size	GZIP Compress	Brotli Compress	GZIP Decompress	Brotli Decompress	Winner
Small (10240 bytes)	101.10ms	99.77ms	21.53ms	23.65ms	Brotli
Medium (102400 bytes)	22.77ms	36.24ms	21.85ms	21.05ms	GZIP
Large (512000 bytes)	23.67ms	38.63ms	22.44ms	24.47ms	GZIP

I did some calculations (I hope they are correct), and I think I can achieve the following goals:

Reduced Costs: decrease cache storage costs by 78% by compressing large objects
Improved Performance: API response times improved by 32% for operations involving large cached objects
Increased Cache Hit Ratio: More data could fit in the same cache size, increasing our hit ratio
Cache Provider Independence: I could use different cache providers without changing the code

I have also found out some interesting facts:

Use GZIP for frequent write operations with medium/large data (faster compression)
Consider Brotli for read-heavy workloads where the compression ratio is critical
Don't compress small objects under 2KB, as the overhead isn't worth it
For batch operations, adjusting compression thresholds based on batch size is beneficial

There are some trade-offs, as usual, to consider:

CPU/Memory Usage: Compression/decompression increased CPU/Memory utilisation, and I could see <5%, but you know it could be important
Complex Logic: The adaptive rules add complexity to the caching layer
Debugging Challenges: Compressed data is not readable for debugging.

Cheating Lambda scalability

Daniele Frasca — Wed, 29 Jan 2025 05:00:00 +0000

I am one of those people that can be defined as serverless first. Almost everything can be serverless, and with some bending, you can find a way when serverless has limitations.

While some argue that it is better to have an architecture with 2 boxes (CLUSTER and Relational Database), I use the features of services to achieve the same results.

For example:

Duplicating data in DynamoDB DB table for better data access vs 1 Relational DB with complex SQL
DynamoDB Streams vs Background Job to find out what has "changed"
Optimising Lambda functions Duration to bring down the cost vs having all in one big cluster
Swapping services when they reach a scale that price makes it no sense, for example, ALB in favour of API Gateway

As code is a liability, with Serverless, I have more control as I keep it to a minimum. If I would build with the Monolith approach of a cluster, I would make the code more "complex" as I would use patterns like:

Multi-tier Architecture or Hexagonal Architecture...
Dependency injection framework
Middleware framework

When I say "complex" with the above options, it is because I now need to leverage some framework, while writing single-responsibility Lambda functions is just a matter of:

Validate the input
Some checks (business logic) with the usage of AWS SDK
Return a response

Now that I have explained why I prefer serverless, I can talk about the limitations of Lambda at scale.

Most people compare only the AWS Bill, and this is not the best way to do it.

There are a lot of good articles out there:

It is not arguable that Lambda can be more expensive per request than a cluster and that we have some hard limits on scalability.

Parameter Management at Scale: How Multi-Region Configurations Saved My Application

Daniele Frasca — Tue, 29 Oct 2024 05:00:00 +0000

In this post, I want to share a story about how significant downtime could have impacted the application if I hadn't used a multi-region setup.

In the last few years, I have undertaken the multi-region road for all my applications. I am leading a new path for a streaming application with a new backend, leveraging AWS's serverless offering. In this article, I do not want to talk about why Serverless is better at the scale of any cluster configuration, why Serverless make it simple to have a multi-region infrastructure or how the total cost of ownership is cheaper than a multi-region cluster infrastructure. This article focuses on something I discovered recently that could cause a major downtime in production.

All my infrastructure is deployed using the AWS Serverless Application Model (AWS SAM). This tool consists of two parts:

AWS SAM templates
AWS SAM CLI

Without going into details, I enable versioning when I use Lambda. Each version makes the Lambda immutable, ensuring that code and configuration do not change once published.

Example:

AWSTemplateFormatVersion: 2010-09-09
Transform:
  - AWS::Serverless-2016-10-31

Mappings:  
  myURL:
    test:
      url: https://test.com
    stage:
      url: https://stage.com
    prod:
      url: https://prod.com

Resources:
  MyFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ../src/handlers/something
      .....
      AutoPublishAlias: live
      Environment:
        Variables:
          SOMETHING: !FindInMap [myURL, !Ref StageName, 'url']
      .....

AWS THEORY

Lambda creates a new version of your function each time that you publish the function. The new version is a copy of the unpublished version of the function. The unpublished version is named $LATEST.
AutoPublishAliasAllProperties - Specifies when a new AWS::Lambda::Version is created. When true, a new Lambda version is created when any property in the Lambda function is modified. When false, a new Lambda version is created only when any of the following properties are modified: Environment, MemorySize, SnapStart or any change that results in an update to the Code property, such as CodeDict, ImageUri, or InlineCode

PROBLEM

Updating only the value of an environment variable (without any other changes in the template/code) can result in a successful deployment that isn't picked up by the lambda function. Because I deployed from CI and saw that everything had been deployed, I mistakenly thought it was all updated. Sadly, no new version exists, and the changes are only applied to $LATEST.

SOLUTION

I think there is only one real solution without considering:

Forcing a new deployment by making a small, inconsequential change to the Lambda function code (why someone will do that)
Adding a parameter to the SAM template that changes with each deployment (it does not work)
Explicitly publish a new version in the deployment pipeline after the SAM deployment. Using the AWS CLI or SDK to do this programmatically (so much effort)

The correct solution is:

AWS::LanguageExtensions transform

I did not know as many people that I asked for, but apparently AWS::LanguageExtensions have been announced back in 2022.

It MUST be used in the correct order:

AWSTemplateFormatVersion: 2010-09-09
Transform: 
  - AWS::LanguageExtensions
  - AWS::Serverless-2016-10-31

With LanguageExtensions, they fixed the previous limitation of being unable to dynamically resolve intrinsic functions or parameter references.

SO WHAT?

Adding LanguageExtensions requires the STACK to be removed.

Removing a stack, especially for APIs on a streaming application with continuous traffic h24, is a recipe for disaster, and the multi-region infrastructure saved me.

On the architecture above, I have:

CloudFront - You can use as API Acceleration with or without cache
Route 53 - As router to create Active-Active or Active-Passive infrastructure
API Gateway - As front regional door. It is possible to use as well ALB or Lambda function URLs
Lambda - as my serverless computational service.

With this architecture, I can remove a stack in Region A without downtime because I only have to shift the traffic with Route 53 to Region B. Once the stack is replaced, I can perform this operation for each region in my architecture until it is all replaced and only then apply reset to the original Route53 configuration.

I wonder if CDK had the same problem, and the answer is YES if you are in some old years version, but of course, the latest did not have it

    const handler = new lambda.Function(this, "handler", {
      code: new Lambda.AssetCode(path.resolve(__dirname, "dist")),
      handler: `index.${config.api.handler}`,
      runtime: Lambda.Runtime.NODEJS_20_X,
      architecture: lambda.Architecture.ARM_64,
      currentVersionOptions: {
        removalPolicy: RemovalPolicy.RETAIN,
        retryAttempts: 1,
      },
      environment: {
        TABLE_NAME: "xxxx",
      },
    });

    const alias = new lambda.Alias(this, 'LambdaAlias', {
      aliasName: 'live',
      version: handler.currentVersion,
    });

    new apigw.LambdaRestApi(this, config.apiName, {
      handler: alias,
      description: config.apiDescription
    });

Conclusion

Since we are all using an SDK to perform our tasks, a similar issues can arise. If the infrastructure is critical, transitioning to a multi-region setup with serverless architecture is quite simple. The pay-per-use model helps keep costs under control, allowing us to pay only for active usage. For instance, an Active-Passive design can help mitigate similar problems without the burden of paying for unnecessary infrastructure.

This stranger EventBus Mesh

Daniele Frasca — Wed, 18 Sep 2024 11:52:09 +0000

During the year 2024, I have been presenting how I integrated a company central bus into a set of serverless distributed services. I have also written some thoughts down, and you can find them here

When I talk about a Microservice architecture I am thinking immediately at:

loosely coupled services
well-defined boundaries
communication through lightweight protocols

When working with Microservices, I usually use Event-driven architecture (EDA). This combination becomes clear when I consider a complex distributed system in which some data is duplicated in multiple services.

For more info, I suggest reading:

Building Event-Driven Architectures

When I design an application, I usually have multiple domains within my boundaries. Sometimes, my services must communicate outside my boundaries, for example, with other teams, and because of this, it is important to distinguish between the types of events:

private/internal events
public/integration events

For more info Internal vs External events

A team can own multiple microservices, each of which can be formed of various components and communicate with each other.
It's important to be aware that microservices that rely on a central bus to communicate their private/internal events can potentially lead to problems.

When a microservice sends events to the entire system, it broadcasts its internal state changes, which can leak business rules and expose its internal state.
This is different from how microservices are meant to work.

Their internal details should not be exposed to other services.
If other services start to rely on these events to function correctly, this can lead to tight coupling between services.

This is problematic because if the service's internal workings change, it can break other services that depend on those events.

The solution is to move to an EventBus Mesh.

Each team has one central bus, which sends events in and out with the company's central bus.
All the services within the team do not know about the existence of the company's central bus. They only know about their local bus and communication within the services is made with the local bus, which routes the event to the team bus.
The key aspect to focus on is setting clear "boundaries".
This mechanism is similar to Pub/Sub, but I subscribe to a Bus instead of an SQS Queue or SNS Topic.
Utilising an EventBusMesh allows me to control the publishing and subscribing of those events while maintaining isolation.
This approach helps to ensure loose coupling between services and enables each service to evolve independently.

The EventBridge Mesh design can be quickly implemented using Amazon EventBridge because it allows configuration of EventBridge to send and receive events between event buses in AWS accounts within the same region, across all regions, and between accounts in different regions as long as the destination region is a supported cross-region destination. When configuring EventBridge to send or receive events between accounts, you can specify which AWS accounts can send to or receive events from the event bus in your account. Additionally, you can allow or deny events based on specific rules associated with the event bus or events from particular sources. This is very important because, without an explicit denial, this design will not block a private event from reaching the company bus. By using IAM policy conditions in Amazon EventBridge I can add a policy to deny specific set of sources and event types.

EventBridge references:

Conclusion

Hopefully, with this post, the EventBus Mesh pattern becomes clearer. By using Amazon EventBridge in combination with Amazon EventBridge Pipes, I gained the ability to connect multiple features and achieve isolation and control when publishing and reading those events. A clear drawback of this pattern is that putting events onto another bus is charged at the same rate as other put events operations. On the other hand, I didn't have to write any code except for the infrastructure configuration required for different types of buses. The extra charge on the Put operation is cheaper than writing your code to achieve similar isolation with other busses, such as Kafka to Kafka.

Integrate Kafka with a Serverless application

Daniele Frasca — Wed, 22 May 2024 04:26:33 +0000

We are all working with the microservices pattern, or at least in a context with multiple teams where information exchange is necessary.

Introduction

In an event-driven-application setup, I usually see 3 main components to exchange messages:

Apache Kafka is an open-source, high-performance, fault-tolerant, and scalable platform for building real-time streaming data pipelines and applications. Streaming data is continuously generated by thousands of data sources, which simultaneously send the data records in.
Amazon EventBridge is a serverless service that uses events to connect application components, making it easier to build scalable event-driven applications. Event-driven architecture builds loosely coupled software systems that work together by emitting and responding to events.
Amazon Kinesis is an AWS version similar to Kafka, a high-performance, fault-tolerant, and scalable platform for building real-time streaming data pipelines and applications. Streaming data is continuously generated by thousands of data sources, which send the data records simultaneously.

While Apache Kafka and Amazon Kinesis share many similarities in terms of functionality, they are not identical. Their choice often depends on factors such as use cases and infrastructure.

In theory, streaming platforms like Kafka and Kinesis Data Streams and event routers like EventBridge serve different purposes and have different characteristics:

Streaming Platforms are designed to handle high-volume real-time data streams. They provide capabilities for storing, processing, and analysing streams of records in real-time. They are often used in cases where large amounts of live data need to be processed and analysed, such as log monitoring, real-time analytics, and IoT data processing.
Event Routers are designed to route events from various sources to different destinations. Event-driven architectures often use them to decouple microservices, orchestrate workflows, or integrate with third-party services.

Both types of systems deal with events or messages, and the difference is subtle. I have often found a reality where those services are used for the same reasons.

In the company where I work, we use Confluent Apache Kafka as a common bus for all teams. Regardless of the architecture, whether it's Cluster or Serverless, they all subscribe to or publish to Kafka.

The direct integration to Kafka requires code for:

Filtering
Transforming the message
Moving to some other downstream service
Handling the error

The problem with writing code is that you must maintain it, which is never the same over time. When the architecture grows, sometimes it gets wild, and there is no longer a standard.

For example, Kafka has the concept of Consumer Groups. A Kafka Consumer Group is a feature that allows a pool of consumers (each with a unique ID) to divide up the processing of data over a collection of Kafka topics.

In simple words, it means this:

Kafka is configured with 3 consumer groups for a Topic
I have 3 workers that subscribe to that Topic

When a worker (consumer) fails to process a message and throws an error, Kafka's default behaviour is to retry delivering the message. If the worker continues to throw an error for the same message, this can lead to repeated retries, blocking the consumer group and creating a lag.

The solution could be for the worker to catch the error and either handle it or ignore it. It can then manually commit the offset for the message, telling Kafka it has been processed. This will prevent Kafka from trying to redeliver the same message or trying to move it to a DLQ.

Even if it is logically easy, I can guarantee that with time, there will be services that do it, some will swallow the error, and so on, creating a mess.

I usually try to mitigate issues I can have while working with distributed systems, so I tend to migrate to or use the AWS Serverless offering.

Integration types

I know I sound like a broken record, but I'll repeat it: the AWS Serverless offering is 2 things:

AWS takes care of scalability and availability and patching of their services
I integrate their managed services and focus on what truly matters: writing code and delivering business value.

When you are working with Apache Kafka in AWS setup, there are 2 types of integrations:

This Kafka sink connector for Amazon EventBridge allows you to send events (records) from one or multiple Kafka topic(s) to the specified event bus, including useful features such as:

offloading large events to S3
configurable Topic to event detail-type name mapping custom IAM profiles per connector
IAM role-based authentication
support for dead-letter queues
schema registry support for Avro and Protocol Buffers (Protobuf).

The connector is currently in version v1.3.0 and has great features compared to Pipes. These are offloading large events to S3 and automatic dead-letter queues, which is a different topic where you can send the failed message without writing code to handle the errors in the consumer logic.

On the other hand, without access to the cluster or relying on a different team for this setup, the best choice to move forward and apply standards to my services is to use Amazon EventBridge Pipes.

EventBridge Pipes is an incredible service that offers point-to-point connections between a source and a target. This feature can be very powerful, especially when we use EB as a target. Pipes handle all point-to-point connections efficiently, allowing us to transform the message and ensure that the downstream consumer can understand it.

There are so many other concepts, but the most exciting part of this service is that it enables me to enrich and modify the message before passing it to the downstream consumer.

If we compare the connector and pipe at a glance in the context of Kafka, the most significant differentiators are DLQ/S3 offloading (until available in Pipes) and Avro/Protobuf on the connector side vs. super simple setup/operations and enrichment with Pipes.

Kafka messages can go up to 32MB, while serverless services like EventBridge/SQS/SNS can handle only a maximum of 256KB while Lambda is up to 6MB
While it seems unrealistic to have such events, having events bigger than 256KB is not unrealistic.

What is the problem with such messages?

This is known as a poisoned message. The Amazon EventBridge open-source connector for Apache Kafka Connect can handle this case because it moves the message to a different Kafka Topic or is configured to offload large payloads to S3 and send the “trimmed-down” event to EventBridge, while with Amazon EventBridge Pipes and other types like Lambda/Cluster, it will block the processing of your messages from the Kafka consumer group until the Pipes retries are exhausted. Because the message cannot be moved to a DLQ (SQS only supported), it will be lost forever without you knowing about it.

A limited solution with EventBridge Pipes is to leverage the enrichment concept. With the Claim Check pattern, we can circumvent this limitation. However, this works only if the message in input is not over the 6MB Lambda payload limit.

It is particularly useful when dealing with large data sets or messages. The pattern works by storing the large message in a data store and then passing a reference to the data to subsequent components. These components can then use the claim check to retrieve the data.

Publishing to Kafka

Until now, I wrote only about how to configure Kafka to subscribe to the message, but I also need to publish, and we have 2 options:

Writing code using the SDK
Kafka proxy via API destinations

Writing code is simple once downloaded from the official SDK library. In this example, I will use KafkaJS



const producer = kafka.producer()

await producer.connect()
await producer.send({
    topic: 'topic-name',
    messages: [
        { key: 'key1', value: 'hello world' },
        { key: 'key2', value: 'hey hey!' }
    ],
})

An alternative is to use Confluent REST Proxy for Apache Kafka, and I can do this without writing any code but only leveraging the Serverless offering using EventBridge API destinations.



  
  
  FROM EB TO KAKFKA


ApiDestinationsTargetRole:

    Type: AWS::IAM::Role

    Properties:

      AssumeRolePolicyDocument:

        Version: '2012-10-17'

        Statement:

          - Effect: Allow

            Principal:

              Service: events.amazonaws.com

            Action: sts:AssumeRole

      Path: /service-role/

      Policies:

        - PolicyName: secretmanager

          PolicyDocument:

            Version: '2012-10-17'

            Statement:

              - Effect: Allow

                Action:

                  - "secretsmanager:DescribeSecret"

                  - "secretsmanager:GetSecretValue"

                Resource: "*"

        - PolicyName: destinationinvoke

          PolicyDocument:

            Version: '2012-10-17'

            Statement:

              - Effect: Allow

                Action:

                  - events:InvokeApiDestination

                Resource:

                  - !GetAtt KafkaProxyApi.Arn

MyApiDestinationFailedDeliveryDLQ:

    Type: AWS::SQS::Queue

MyApiDestinationFailedDeliveryDLQPolicy:

    Type: AWS::SQS::QueuePolicy

    Properties:

      Queues:

        - !Ref MyApiDestinationFailedDeliveryDLQ

      PolicyDocument:

        Statement:

        - Effect: Allow

          Principal:

            Service: events.amazonaws.com

          Action: SQS:SendMessage

          Resource: !GetAtt MyApiDestinationFailedDeliveryDLQ.Arn

ApiConnection:

    Type: AWS::Events::Connection

    Properties:

      AuthorizationType: BASIC

      AuthParameters:

        BasicAuthParameters:

          Password: '{{resolve:secretsmanager:confluent_cloud/service_account/myservice:SecretString:password}}'

          Username: '{{resolve:secretsmanager:confluent_cloud/service_account/myservice:SecretString:username}}'

KafkaProxyApi:

    Type: AWS::Events::ApiDestination

    Properties:

      ConnectionArn: !GetAtt ApiConnection.Arn

      HttpMethod: POST

      InvocationEndpoint:

        Fn::Sub:

          - '${RegistryUrl}/kafka/v3/clusters/${ClusterId}/topics/${Topic}/records'

          - RegistryUrl: '{{resolve:secretsmanager:confluent_cloud/service_account/myservice:SecretString:schema_registry_url}}'

            ClusterId: '{{resolve:secretsmanager:confluent_cloud/service_account/myservice:SecretString:cluster_id}}'

            Topic:

              Fn::FindInMap:

                - KafkaTopic

                - Ref: StageName

                - MyApiDestinationModified

      InvocationRateLimitPerSecond: !FindInMap [Api, !Ref StageName, 'invocationRateLimitPerSecond']

MyApiDestinationAddedRule:

    Type: AWS::Events::Rule

    Properties:

      EventBusName: !Ref LocalBusName

      EventPattern:

        source:

          - "mysource"

        detail-type:

          - "SOMETHING_CREATED"

      State: ENABLED

      Targets:

        - Id: MyApiDestination-added

          Arn: !GetAtt KafkaProxyApi.Arn

          RoleArn: !GetAtt ApiDestinationsTargetRole.Arn

          InputTransformer:

            InputPathsMap:

              prop1: $.detail.data.prop1

              prop2: $.detail.data.prop2

              createdAt: $.detail.data.createdAt

            InputTemplate: |-

              {

                "key": {

                  "type": "JSON",

                  "data": "<prop2>"

                },

                "value": {

                  "type": "JSON",

                  "data": {

                    "prop1":  "<prop1>",

                    "prop2":  "<prop2>",

                    "_timestamp": <createdAt>,

                    "action": "CREATED"

                  }

                }

              }

          HttpParameters:

            HeaderParameters:

              Content-Type: "application/json"

          DeadLetterConfig:

            Arn: !GetAtt MyApiDestinationFailedDeliveryDLQ.Arn

          RetryPolicy:

            MaximumRetryAttempts: 3

            MaximumEventAgeInSeconds: 60

Conclusion

In this article, I discussed two types of integration with Apache Kafka and how we can integrate into the AWS ecosystem without giving up the convenience of using managed services. At the time of writing, Kafka EventBridge Connector and Amazon EventBridge Pipes have some differences, which can significantly impact architectural decisions. With time, they will become more equal, but until then, it is crucial to understand the specific use case and the tradeoffs each configuration can bring. Ultimately, I presented two ways to publish messages into Kafka, and again, it is crucial to understand the limitations. For example, using the HTTP API Destination can lead to quota limitations at both the AWS and Confluent Kafka levels. While these limitations are sometimes flexible, writing simple code in highly concurrency scenarios may be easier.

Serverless Challenge: Is scaling achievable using Momento as a cache-aside pattern?

Daniele Frasca — Thu, 14 Dec 2023 05:00:00 +0000

This post has been delayed because of re:Invent2023. Could some announcements impact my architectural decision?

I am talking about the Amazon ElastiCache Serverless for Redis.
AWS is empowering us with their services; this Redis serverless version is a testament to that.
It may be a partial serverless solution but is a step toward a more efficient and seamless experience.

At first glance, I noticed:

Base price of $90/month/GB + 0.34 cents/GB of data transferred
The capability to autoscale every 10 minutes is not enough for spiky traffic
There is a VPC

After verifying the information, I knew that Momento was the better choice. Momento was introduced in November 2022, and I closely followed them. My application requires caching of nearly everything to safeguard the downstream sources, including DynamoDB and S3.

I have approached Momento explaining my need for caching, and in particular, we looked into a specific case when I needed to cache item sizes over 500KB up to 2MB.
During my previous tests with S3 and Redis, I encountered scalability issues which I wanted to minimise as much as possible.

My load test is always the same:

Item cached in the CDN for 3 minutes
Test time 2h
Total requests ~200M
RPS > 10K up to 30K

Each request can load N keys up to 20 or 30, and those references are stored in DynamoDB. If I reach 10K RPS with my APIGW, I could hit the cache with 200K or 300K RPS and at 30k RPS of traffic, almost 1M RPS.

Here is a recent spike with raw objects up to 1MB in size (compression makes it around 100KB)

I have achieved remarkable heights without glitches, bringing the error count down to zero and making Momento an outstanding partner you can always rely on.

The test results were quite revealing, showing two areas of significant improvement:

speed
cost

SPEED

At first, I encountered unexpected speed issues, and the results were slower than expected due to the cache size and cross-region access. This access added 20ms to each request, but I could remove it when in the same region.
The payload size was a significant challenge, and I knew compression was the solution. Although my initial Rust implementation was unsuccessful, the Momento team stepped in with their expertise and implemented data compression into the SDK, making the solution even more effective.

Momento's expert use of the Zstandard algorithm has resulted in my data being compressed by 90% without any compromise on speed. The latency is comparable to having Redis in the VPC, making data processing faster and more efficient.

The code in Rust to SET/GET a batch of keys in parallel using the compression looks like this:

use crate::dtos::page_item_response::CacheItem;
use async_trait::async_trait;
use momento::response::Get;
use std::sync::Arc;
use tracing::{error, warn};
use typed_builder::TypedBuilder;

#[cfg(test)]
use mockall::{automock, predicate::*};

#[cfg_attr(test, automock)]
#[async_trait]
pub trait CacheService {
    async fn get_by_keys(&self, keys:&[String]) -> anyhow::Result<Vec<CacheItem>>;
    async fn set_by_keys(&self, items: &[CacheItem]) -> anyhow::Result<()>;
}

#[derive(TypedBuilder)]
pub struct Momento {
    cache_name: Arc<String>,
    client: Arc<momento::SimpleCacheClient>,
}

#[async_trait]
impl CacheService for Momento {
    async fn get_by_keys(&self, keys: &[String]) -> anyhow::Result<Vec<CacheItem>> {
        let tasks = keys.iter().map(|key| {
            let key: Arc<String> = Arc::from(key.clone());
            let mut shared_client = (*self.client).clone();
            let cache_name = self.cache_name.clone();

            tokio::spawn(async move {
                let response = shared_client
                    .get_with_decompression(&cache_name, key.as_bytes())
                    .await
                    .ok();

                match response {
                    Some(Get::Hit { value }) => {
                        let value: String = value.try_into().ok()?;
                        serde_json::from_str::<CacheItem>(&value).ok()
                    }
                    Some(Get::Miss) => {
                        warn!("MISSING CACHE KEY: {}", key);
                        None
                    }
                    None => {
                        error!("MOMENTO get is null {}", key);
                        None
                    }
                }
            })
        });

        Ok(futures::future::join_all(tasks)
            .await
            .into_iter()
            .filter_map(|result| result.ok().flatten())
            .collect())
    }

    async fn set_by_keys(&self, items: &[CacheItem]) -> anyhow::Result<()> {
        let tasks = items.iter().map(|item| {
            let item_cloned = item.clone();
            let mut shared_client = (*self.client).clone();
            let cache_name = self.cache_name.clone();

            tokio::spawn(async move {
                let cache_item = CacheItem::builder()
                    .value(item_cloned.value)
                    .id(item_cloned.id)
                    .cache_key(item_cloned.cache_key)
                    .build();
                let json = serde_json::to_string(&cache_item).unwrap();
                shared_client
                    .set_with_compression(&cache_name, cache_item.cache_key.clone(), json, None)
                    .await
                    .map_err(|e| error!("ERROR - Momento - set_object {:?}", e))
                    .ok();
            })
        });

        futures::future::join_all(tasks).await;
        Ok(())
    }
}

COST

Momento pricing is simple:

They charge for $0.50 per GB transferred
Allowing up to 1M+ Operations Per Second (based on your profile)

If I take my Redis cluster NetworkBytesOut metric per month aggregation, I could have a rough cost:

XTB: $0.50/GB * X,000GB = $xxx

Considering the active compression option, the price can be reduced by about 90%.

Calculating the cost for a similar load using Redis is not straightforward. When operating inside a VPC, there are additional things to consider, such as the Data Processing of NAT Gateways, and the Data Transfer to the Internet, if any.

It's important to note that comparing costs is not as simple as comparing apples to apples. Before drawing any conclusions, I would suggest to get in touch with Momento. They offer significant workflow discounts that can make the cost highly competitive.

In addition to the Serverless Cache that I can use with a Lambda without a VPC, the Momento Cache system offers different connectivity options that can be used inside the VPC, including:

VPC Peering
Private Link
NAT Gateway
Optional Zone Aware Endpoints

From a tenancy perspective, they can offer both shared tenancy as well as dedicated tenancy with varying levels of isolation.

Momento is not just a cache. It offers support for topics which Redis does not. With Redis, to use pub-sub, one needs to configure WebSockets, such as an API gateway or a web server, and then add a layer for authentication. However, with Momento, all of these features are built within a single platform. I can publish and subscribe to topics in a browser using the Momento Web SDK. When data is published to a topic, all subscribers will receive the data, which means browsers can be connected without building server-side code.

Momento is a true example of a Serverless SaaS service because it offers powerful services, extreme time to market, and reliability. I compare it to services like Lambda regarding power and time to market.

Conclusion

I have found that using compression can significantly reduce costs by up to 90%. While using Momento over Redis resulted in similar p99 latency, the scalability achieved during testing was impressive. Moreover, the cache system did not cause any errors or overflow downstream services, which has given me confidence in my choice. With a generous free tier, an on-demand pricing model that reduces waste between environments and traffic peaks, and the ability to get up and running quickly with no infrastructure setup or maintenance makes Momento a compelling solution for my workload.

re:Invent 2023

Daniele Frasca — Sun, 03 Dec 2023 16:46:50 +0000

Regarding re:Invent 2023, it was my second year in the raw and was terrific.

I am part of a fantastic community, the AWS Community Builders

The AWS Community Builders program offers technical resources, education, and networking opportunities to AWS technical enthusiasts and emerging thought leaders who are passionate about sharing knowledge and connecting with the technical community.
Interested AWS builders should apply to the program to build relationships with AWS product teams, AWS Heroes, and the AWS community.
Throughout the program, AWS subject matter experts will provide informative webinars, share insights — including information about the latest services — as well as best practices for creating technical content, increasing reach, and sharing AWS knowledge across online and in-person communities. The program will accept a limited number of members per year. All AWS builders are welcome and encouraged to apply.

For me, it is all about building relationships, and this year, AWS provided us with the Community Hub.

In partnership with the Modern Applications Community we're doing a four day buy-out of Buddy V's Ristorante, located conveniently inside the Venetian, where we'll be hosting an all-encompassing experience exclusively for our program members. Whether you're a Hero, Community Builder, and/or a User Group Leader, you'll be able to join us at the Hub Monday through Thursday during the day. You'll have opportunities to meet, mingle, get food/drinks, and network with each other and service team members during re:Invent week.

It is incredible how much we can learn from simply engaging with others. I find great pleasure in conversing with people, particularly when discussing Serverless. Through the AWS Community Builder program, I have had the opportunity to share my experiences and connect with individuals who have encountered similar obstacles. The Community Hub has provided an exceptional environment to fully engage with the Heroes, Builders, and service team. I never expected to be recognized or to have others mention that they follow me on Twitter or have seen me speak. But what was truly remarkable was my sense of belonging and connection. This community has inspired me to share my knowledge and travel throughout Europe to spread my passion for Serverless.

I describe myself as a person who loves interacting with people and thrives on tackling challenges (also shy). I was determined to create something unique and extraordinary to set me apart from the regular "Hello World" enthusiasts when I joined the program. My passion for using AWS Serverless services has led me to explore complex scenarios where traditional approaches don't cut it. Whether writing an article or speaking at a conference, I always strive to share my real-life experiences, highlighting the limitations and possible solutions to the problems I've faced. I constantly face challenges but enjoy them and always strive to overcome them.

This year's event was an eye-opener in many ways, revealing exciting new insights I can't wait to explore further!

LLRT - was a great DEMO
S3 Express One Zone storage class
Amazon DynamoDB zero-ETL integration with Amazon OpenSearch Service
AWS Lambda functions now scale 12 times faster when handling high-volume requests
AWS Lambda introduces new controls to make it easier to search, filter, and aggregate Lambda function logs
AWS SDK for Rust is now generally available
Kafka and EventBridge better together

I want to express my gratitude to all the wonderful people who spent time chatting with me, laughing alongside me (or even at me!), and making me feel included.

You all are amazing! I'm looking forward to seeing you again soon.

Your friendly AWS Community Builder.

Forem: Daniele Frasca

Three Responsibilities of a Global Application (Part 2)

Why global systems feel complex

Responsibility #1 — Doing the work (execution)

Responsibility #2 — Proving the work (evidence and control)

Responsibility #3 — Understanding the system (operations)

Why mixing responsibilities breaks systems

Benefits of separation

Conclusion

Trust Is the Architecture (Part 1)

When trust fails

1) Trust between teams

2) Trust between systems - EDA

3) Trust across boundaries - data residency

4) Trust with external - auditors, regulators, customers

Conclusion

Building long-running usage reports on AWS

Table of contents

The flow

1) Request contract

2) CreateOrGetReport Lambda

CreateOrGetReport (handler)

3) Athena aggregation

Athena completion events

4) Metrics materialisation

5) Queueing narrative generation

Implementation A: Step Functions

A1) Step Functions state machine

A2) Athena resume handler

1) RegisterAthenaWait Lambda

2) EventBridge handler (Athena completion → SendTaskSuccess)

A3) SQS worker (Step Functions)

Worker code

Implementation B: Lambda Durable Functions

B1) Durable orchestrator

Orchestrator

B2) Athena resume handler (Durable)

B3) SQS worker (Durable)

Cost comparison

What is actually different?

Conclusion

When (Not If) Containers Misbehave

The Hidden Cost Equation

What Makes Fargate "Complex"

Deployment Strategy

Resource Allocation: CPU and Memory Sizing

Fargate Cold Start

The 14 potential pitfalls

Performance trade-offs

It's Not About Cost, It's About Risk

What's Next: Traffic Controller

Fargate + Lambda are better together

NOTE

The cool part?

Is Fargate really cheaper than Lambda?

What's Next: When (Not If) Containers Misbehave

Adaptive Compression for Cache Systems with Hexagonal Architecture

Cheating Lambda scalability

Parameter Management at Scale: How Multi-Region Configurations Saved My Application

AWS THEORY

PROBLEM

SOLUTION

SO WHAT?

Conclusion

This stranger EventBus Mesh

Conclusion

Integrate Kafka with a Serverless application

Introduction

Integration types

Publishing to Kafka

FROM EB TO KAKFKA

Conclusion

Serverless Challenge: Is scaling achievable using Momento as a cache-aside pattern?

SPEED

COST

Conclusion

re:Invent 2023