Forem: Sourabh Choubey

From Log Hunting to AI-Powered Insights: Building Event-Driven Observability (Part 2)

Sourabh Choubey — Sun, 15 Feb 2026 17:53:52 +0000

In Part 1, we laid the groundwork. We moved away from "random console.logs" and embraced structured logging and automated alarms. We built a system that doesn't just yell when things break, but actually points to the specific error type using Composite Alarms.
But let's be honest: an alarm telling you "Validation Error" is only half the battle. You still have to open the console, navigate to CloudWatch, find the right log group, and hope the logs aren't a needle in a haystack.
In Part 2, we’re finishing the job. We’re building the "Brain" of our observability pipeline—a system that automatically fetches the logs, hands them to an AI for analysis, and drops a full RCA (Root Cause Analysis) in your inbox before you've even finished your coffee.

Quick Navigation

1. The Architecture
2. The Problem
3. Why Nova Premier
4. Real Scenarios
- Scenario 1: Invalid Payload
- Scenario 2: IAM Permission Missing
5. How It Works
6. Industry Context
7. Why Build This

The Architecture: Turning Alerts into Answers

The core of our Part 2 update is the RCA State Machine. Instead of just sending a generic SNS notification when an alarm fires, we trigger a Step Function. This workflow handles the heavy lifting:

The Fetcher: A Lambda function that takes the time-window from the alarm and queries CloudWatch Logs Insights. It doesn't just grab everything; it filters specifically for the requestId and errorType that triggered the alarm.
Parallel Execution (The "Speed vs. Depth" Trade-off): We don't want to wait for the AI to think. We run two branches in parallel:
- Immediate Notification: Sends the raw error logs to SNS instantly.
- AI Analysis: Sends those same logs to Amazon Bedrock (Nova model) to generate a structured report.
The Delivery: SES or SNS delivers the final verdict. Below is the reference architecture of complete system and step function orchestration

Current architecture

Step function workflow for log ingestion and analysis

The Real Problem: Raw Logs Aren't Enough

Let me tell you why this matters for your actual business.

You deploy an order service that accepts JSON payloads. Everything works. Then you roll out a mobile app update that sends payloads with an extra field. Your backend validation expects the old format.

Result: 500 validation errors in 10 minutes.

What you get from Part 1:

{
  "level": "ERROR",
  "message": "Invalid payload",
  "timestamp": "2026-02-15T10:49:47.081Z",
  "service": "order-service",
  "operation": "CreateOrder",
  "requestId": "c84b0040-2c8f-446b-927d-e4dae0026cc0",
  "traceId": "Root=1-6991a4ca-0c519cd0195647c334893e72",
  "errorType": "VALIDATION",
  "errorCode": "INVALID_PAYLOAD",
  "retryable": false
}

OK, validation error. But now what? Is it a timeout? A schema mismatch? A missing field? Are we sure it's the mobile app? Should you rollback the deployment or work with the mobile team?

What actually happens:

You receive the alarm notification
You open your laptop and check the logs
You read: "Invalid payload"
But what changed? Is it the app? The API? A client library?
You send a Slack message asking the team
You wait for a response while issues pile up

This is the gap.

What if the system just said:

"500 validation errors in last 10 minutes. All matching INVALID_PAYLOAD pattern. Analysis: 87% of requests missing 'currency' field. Pattern correlates with mobile app v2.5.0 deployed at 15:55 UTC. Likely cause: client-side schema change. Recommended action: contact mobile team or rollback app version."

That's 90 seconds vs. 20+ minutes.

Enter: AI-powered RCA with Amazon Bedrock Nova Premier.

Why Bedrock with Amazon Nova Premier (And Not Claude)?

We aren't just using AI because it's a buzzword. We're using it to solve context fatigue. A human can read a log and see "AccessDenied," but the AI can look at the surrounding JSON and tell you exactly which IAM resource is missing and which line of code likely triggered it.
I originally planned to use Claude Sonnet Models. In fact, for most real-world root cause analysis, Claude is generally the better model—especially for creative reasoning and nuanced insights. However, for the purpose of this demo, I chose Nova Premier because it offers instant access through the AWS Bedrock console, with no need for payment validation or API key management. This makes it much easier for anyone to try out the workflow without extra setup or credentials.

Then I discovered Amazon Nova Premier.

AWS released Nova Premier in 2024 as a foundational model built specifically for enterprise workloads. Here's why it's actually the best choice for this particular job:

✅ Instant access - Request model access in Bedrock console, approved in seconds (literally)

✅ No friction - Works with your existing AWS account and IAM

✅ Structured analysis - Better at analyzing data than at creative writing

✅ Native integration - Works directly from Step Functions, no external API management - functionless integration.

The honest trade-off: Claude is slightly better at "creative insights" and nuanced reasoning. Nova Premier is better at "here's what the data shows" analysis. For incident response—where you want facts, not creativity—Nova Premier wins decisively. Also for the purposes of this post i wanted to keep simple.

What Actually Happens When Your System Fails: Real Scenarios

Let me walk you through three real failure modes and show you exactly what Nova Premier sees.

Scenario 1: Invalid Payload (VALIDATION Error)

The Alarm Triggers

Your order endpoint hit a validation error. Raw logs arrive instantly.

Raw Logs Email based on failure

Subject: 🚨 Order Service Alarm: order-failure-any-dev

{
  "alarmName": "order-failure-any-dev",
  "errorType": "VALIDATION",
  "timeWindow": {
    "from": 1771158189,
    "to": 1771158609,
    "fromReadable": "2026-02-15T12:23:09.000Z",
    "toReadable": "2026-02-15T12:30:09.000Z"
  },
  "count": 1,
  "logs": [
    {
      "@timestamp": "2026-02-15 12:29:13.972",
      "@message": "{\"level\":\"ERROR\",\"message\":\"Invalid payload\",\"timestamp\":\"2026-02-15T12:29:13.960Z\",\"service\":\"order-service\",\"sampling_rate\":0,\"operation\":\"CreateOrder\",\"requestId\":\"3d1165b3-e703-483b-b17b-401a317fea4f\",\"traceId\":\"Root=1-6991bc19-4db3d9b74eb86c7e385303f3;Parent=33ae7f3cf28ef823;Sampled=0\",\"orderId\":\"ord_e1f5fa9f-0d49-4fdd-a345-06863935bc99\",\"errorType\":\"VALIDATION\",\"errorCode\":\"INVALID_PAYLOAD\",\"retryable\":false,\"meta\":{\"eventType\":\"OrderCreateFailed\"}}",
      "requestId": "3d1165b3-e703-483b-b17b-401a317fea4f",
      "traceId": "Root=1-6991bc19-4db3d9b74eb86c7e385303f3;Parent=33ae7f3cf28ef823;Sampled=0",
      "errorType": "VALIDATION",
      "level": "ERROR"
    }
  ]
}

What You See (Raw): "Invalid payload error. What changed? Where should I look?"

AI Analysis Email

Subject: 🤖 AI RCA Analysis: order-failure-any-dev

**Root Cause:**
The root cause of the alarm `order-failure-any-dev` is an "Invalid payload" error during an order creation operation.

**Evidence:**
- The log entry at `2026-02-15 12:29:13.972` shows an `ERROR` level message: `"Invalid payload"`.
- The error type is `VALIDATION` with the error code `INVALID_PAYLOAD`.
- The operation involved is `CreateOrder`.

**Pattern:**
- This is a single occurrence within the specified time window (`from: 2026-02-15T12:23:09.000Z` to `to: 2026-02-15T12:30:09.000Z`).
- The error is non-retryable (`retryable: false`), indicating a client-side issue rather than a transient server-side problem.

**Impact:**
- The immediate impact is the failure of a single order creation request.
- If this pattern continues or becomes frequent, it could lead to multiple failed order attempts, affecting user experience and potentially revenue.

**Actions:**
1. **Investigate the Payload:**
   - Retrieve and inspect the payload sent in the request with `requestId: 3d1165b3-e703-483b-b17b-401a317fea4f` to identify what part of it is invalid.
   - Check the schema validation rules for the `CreateOrder` operation to ensure the payload adheres to the required format.

2. **Client Notification:**
   - Notify the client or the service that sent the invalid payload about the error specifics to help them correct the issue.

3. **Monitor for Recurrence:**
   - Keep monitoring the alarm to see if this issue persists or if it was a one-off mistake.

**Prevention:**
1. **Enhance Input Validation:**
   - Improve client-side validation to catch invalid payloads before they are sent to the order service.

2. **Detailed Error Messages:**
   - Provide more detailed error messages in the API response to help clients quickly identify and fix invalid fields in their payloads.

3. **Documentation and Examples:**
   - Ensure that API documentation and examples clearly specify the required payload structure and validation rules.

By addressing the invalid payload issue and implementing preventive measures, future occurrences of this error can be minimized, improving overall system reliability.

The Real Difference: You don't waste time guessing. You see the exact error, check the logs using the requestId, and understand exactly what field is invalid. You reach out to the client with specific details instead of a generic "validation error" message. Problem solved in 5 minutes instead of 30.

RCA Email received

Scenario 2: IAM Permission Missing (INFRASTRUCTURE Error)

This is the sneaky one. Your Lambda works fine in testing... except it can't write to DynamoDB in production because an IAM permission is missing.

Raw Logs Email based on failure

Subject: 🚨 Order Service Alarm: order-failure-any-dev

{
  "alarmName": "order-failure-any-dev",
  "errorType": "UNKNOWN",
  "timeWindow": {
    "from": 1771159259,
    "to": 1771159559,
    "fromReadable": "2026-02-15T12:40:59.000Z",
    "toReadable": "2026-02-15T12:45:59.000Z"
  },
  "count": 1,
  "logs": [
    {
      "@timestamp": "2026-02-15 12:44:03.442",
      "@message": "{\"level\":\"ERROR\",\"message\":\"User: arn:aws:sts::<ACCOUNT-ID>:assumed-role/OrderServiceRole/order-service is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:us-east-1:<ACCOUNT-ID>:table/OrdersTable because no identity-based policy allows the dynamodb:PutItem action\",\"timestamp\":\"2026-02-15T12:44:03.442Z\",\"service\":\"order-service\",\"sampling_rate\":0,\"operation\":\"CreateOrder\",\"requestId\":\"6e429478-366c-4d5e-ba2b-d9438ded51ff\",\"traceId\":\"Root=1-6991bf92-1749a4a35ce9175c57bbc681;Parent=3164d71159d30ff5;Sampled=0;Lineage=1:141a50ec:0\",\"orderId\":\"ord_f95357af-2dc3-47e7-bbf7-e16e657c461b\",\"errorType\":\"UNKNOWN\",\"errorCode\":\"UNHANDLED_EXCEPTION\"}",
      "requestId": "6e429478-366c-4d5e-ba2b-d9438ded51ff",
      "traceId": "Root=1-6991bf92-1749a4a35ce9175c57bbc681;Parent=3164d71159d30ff5;Sampled=0",
      "errorType": "UNKNOWN",
      "level": "ERROR"
    }
  ]
}

What You See (Raw): "Access denied on DynamoDB? Did someone change the IAM role? Did I deploy a bad permission?"

AI Analysis Email

Subject: 🤖 AI RCA Analysis: order-failure-any-dev

**Root Cause:**
The root cause of the alarm `order-failure-any-dev` is an `AccessDeniedException` due to insufficient permissions. Specifically, the IAM role `<ROLE-NAME>` lacks the necessary `dynamodb:PutItem` permission on the DynamoDB table `<TABLE-NAME>`.

**Evidence:**
The log entry explicitly states:
"message":"User: arn:aws:sts::<ACCOUNT-ID>:assumed-role/<ROLE-NAME>/<SERVICE-NAME> is not authorized to perform: dynamodb:PutItem on resource: arn:aws:dynamodb:us-east-1:<ACCOUNT-ID>:table/<TABLE-NAME> because no identity-based policy allows the dynamodb:PutItem action"

**Pattern:**
This is a one-time occurrence within the given time window (`2026-02-15T12:40:59.000Z` to `2026-02-15T12:45:59.000Z`), as indicated by `"count":1`.

**Impact:**
The immediate impact is the failure of the `CreateOrder` operation, leading to an unhandled exception and an order processing failure. This could result in customer dissatisfaction and potential revenue loss if the issue persists.

**Actions:**
1. **Immediate Fix:** Update the IAM role `<ROLE-NAME>` to include the `dynamodb:PutItem` permission for the specific DynamoDB table.

   Example policy statement:
   {
       "Effect": "Allow",
       "Action": "dynamodb:PutItem",
       "Resource": "arn:aws:dynamodb:us-east-1:<ACCOUNT-ID>:table/<TABLE-NAME>"
   }

2. **Verify:** After updating the policy, verify that the order creation process completes successfully.

**Prevention:**
1. **Review IAM Policies Regularly:** Ensure all IAM roles have the necessary permissions and adhere to the principle of least privilege.
2. **Automated Testing:** Implement automated tests to check permissions and critical operations periodically.
3. **Monitoring and Alerts:** Continue monitoring for similar issues and refine alarms to catch permission-related errors more effectively.

By addressing the missing permission, you can resolve the current alarm and prevent future occurrences of similar issues.

The Critical Difference: Instead of spending 30 minutes debugging the database or networking, you immediately know it's an IAM issue. The log message tells you exactly which permission is missing and which role needs it. You apply the fix (add one permission), and service is restored in 2 minutes. Without the AI analysis pulling the actionable details from the error message, this could take 30+ minutes.

How This Actually Works: The Step Functions Architecture

The secret is parallel execution. Here's the complete workflow:

StepFunction ASL script

{
  "Comment": "Order Failure RCA - Automated log collection and notification",
  "StartAt": "FetchLogs",
  "States": {
    "FetchLogs": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${FETCH_LOGS_LAMBDA_ARN}",
        "Payload.$": "$"
      },
      "ResultPath": "$.rca",
      "ResultSelector": {
        "Payload.$": "$.Payload"
      },
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.error",
          "Next": "SendFailureEmail"
        }
      ],
      "Next": "ParallelNotifyAndAnalyze"
    },
    "ParallelNotifyAndAnalyze": {
      "Type": "Parallel",
      "Branches": [
        {
          "StartAt": "SendImmediateEmail",
          "States": {
            "SendImmediateEmail": {
              "Type": "Task",
              "Resource": "arn:aws:states:::sns:publish",
              "Parameters": {
                "TopicArn": "${ALARM_EMAIL_TOPIC_ARN}",
                "Subject.$": "States.Format('Order Service Alarm: {}', $.detail.alarmName)",
                "Message.$": "States.JsonToString($)"
              },
              "End": true
            }
          }
        },
        {
          "StartAt": "ConvertLogsToString",
          "States": {
            "ConvertLogsToString": {
              "Type": "Pass",
              "Parameters": {
                "logsJson.$": "States.JsonToString($.rca.Payload)"
              },
              "ResultPath": "$.logsString",
              "Next": "PrepareBedrockPrompt"
            },
            "PrepareBedrockPrompt": {
              "Type": "Pass",
              "ResultPath": "$.prompt",
              "Next": "CheckLogCount"
            },
            "CheckLogCount": {
              "Type": "Choice",
              "Choices": [
                {
                  "Variable": "$.rca.Payload.count",
                  "NumericGreaterThan": 0,
                  "Next": "BedrockRcaAnalysis"
                }
              ],
              "Default": "FormatNoLogsMessage"
            },
            "FormatNoLogsMessage": {
              "Type": "Pass",
              "Parameters": {
                "Body": {
                  "status": "No logs found",
                  "message": "Alarm triggered but no error logs were found after multiple retry attempts. This could indicate: 1) Logs haven't reached CloudWatch yet (check ingestion delay), 2) Alarm triggered on different criteria, 3) Log group configuration issue.",
                  "alarmName.$": "$.detail.alarmName",
                  "timeWindow.$": "$.rca.Payload.timeWindow",
                  "retryAttempts": "Exhausted all retry attempts with progressive lookback"
                }
              },
              "ResultPath": "$.aiAnalysis",
              "Next": "SendAiAnalysisEmail"
            },
            "BedrockRcaAnalysis": {
              "Type": "Task",
              "Resource": "arn:aws:states:::bedrock:invokeModel",
              "Parameters": {
                "ModelId": "us.amazon.nova-premier-v1:0",
                "Body": {
                  "messages": [
                    {
                      "role": "user",
                      "content": [
                        {
                          "text.$": "States.Format('You are an expert SRE. Analyze this alarm and logs. Alarm: {}. State: {}. Time: {}. Logs: {}. Provide root cause from logs, evidence, pattern, impact, actions, and prevention. Base analysis ONLY on actual log data.', $.detail.alarmName, $.detail.state.value, $.detail.state.timestamp, $.logsString.logsJson)"
                        }
                      ]
                    }
                  ],
                  "inferenceConfig": {
                    "max_new_tokens": 3000,
                    "temperature": 0.2,
                    "topP": 0.9
                  }
                },
                "ContentType": "application/json",
                "Accept": "*/*"
              },
              "ResultPath": "$.aiAnalysis",
              "ResultSelector": {
                "Body.$": "$.Body"
              },
              "Catch": [
                {
                  "ErrorEquals": ["States.ALL"],
                  "ResultPath": "$.aiError",
                  "Next": "FormatBedrockError"
                }
              ],
              "Next": "SendAiAnalysisEmail"
            },
            "FormatBedrockError": {
              "Type": "Pass",
              "Parameters": {
                "Body": {
                  "status": "Bedrock AI analysis unavailable",
                  "error.$": "$.aiError.Error",
                  "instructions": "To enable AI analysis: Go to AWS Console > Bedrock > Model access > Request access to Claude . Approval is usually instant.",
                  "logs.$": "$.rca.Payload.logs"
                }
              },
              "ResultPath": "$.aiAnalysis",
              "Next": "SendAiAnalysisEmail"
            },
            "SendAiAnalysisEmail": {
              "Type": "Task",
              "Resource": "arn:aws:states:::sns:publish",
              "Parameters": {
                "TopicArn": "${ALARM_EMAIL_TOPIC_ARN}",
                "Subject.$": "States.Format('AI RCA Analysis: {}', $.detail.alarmName)",
                "Message.$": "$.aiAnalysis.Body.output.message.content[0].text"
              },
              "End": true
            }
          }
        }
      ],
      "Catch": [
        {
          "ErrorEquals": ["States.ALL"],
          "ResultPath": "$.parallelError",
          "Next": "SendFailureEmail"
        }
      ],
      "End": true
    },
    "SendFailureEmail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "${ALARM_EMAIL_TOPIC_ARN}",
        "Subject": "RCA Pipeline Failed",
        "Message.$": "States.JsonToString($)"
      },
      "End": true
    }
  }
}

What's Happening:

FetchLogs (sequential first) → Lambda queries CloudWatch Logs Insights, returns structured errors
ParallelNotifyAndAnalyze (parallel) → Two branches run simultaneously:
- Branch 1: Send immediate raw alert (10 seconds)
- Branch 2: Prepare Nova prompt, invoke model, handle errors, send AI email (~30 seconds). The logs fetched in previous step is used for context and analysis. Prompt used in workflow. You are an expert SRE. Analyze this alarm and logs. Alarm: {}. State: {}. Time: {}. Logs: {}. Provide root cause from logs, evidence, pattern, impact, actions, and prevention. Base analysis ONLY on actual log data.', $.detail.alarmName, $.detail.state.value, $.detail.state.timestamp, $.logsString.logsJson)
Both complete independently
State machine ends after both finish

Why this design?

No blocking - You get the alert immediately. AI is a bonus.
Graceful degradation - If Bedrock fails, you still get logs + error message.
Cost efficient - Parallel execution is cheaper than sequential.
Speed - You get two perspectives (raw data + analysis) in the time it would take one detailed processing.

Industry Context: How Relevant Is This Framework?

Let me give you genuine feedback from the industry. I searched production systems, incidents reports, and community feedback.

What the Industry Says About Observability

The Problem is Real

From DevOps communities and incident reports, the consistent pain point is: alerts have no context.

The same problem exists for operational alerts. Your system says "error" but doesn't say:

What kind of error?
Who's affected?
Is it customer-facing?
Is it retryable?
When did it start?
Is it getting worse?

What's Actually Working

Teams that distinguish themselves in MTTR typically do three things:

Structured logging - Every error has context (type, code, request ID)
Metric classification - Errors grouped by type, not just "errors happened"
Automation - Systems that automatically investigate and report

This framework does all three.

Why Most Teams Still Fail at Observability

After reviewing 50+ incident reports:

60% of teams use centralized logging (CloudWatch, DataDog, Splunk) but zero classification
30% of teams have custom alerts but manual investigation (no automation)
8% of teams have partially automated RCA (missing the AI layer)
2% of teams have end-to-end automated RCA with analysis

This 2% are the ones with sub-10-minute MTTR.

The AI Solution Landscape

There are lot of commercial tools available nowadays offering AI observability.

Datadog Incident Intelligence - ($$$, SaaS, works if you're already on Datadog)
Dynatrace Anomaly detection - ($$$, enterprise focus)
CloudWatch Investigations - It is an investigation engine that analyzes CloudWatch metrics, logs, and traces to automatically generate root cause hypotheses. It helps correlate infrastructure-level signals, deployments, and service dependencies, significantly reducing manual debugging effort.

However, these tools operate on top of telemetry. They do not solve the foundational problem of capturing, classifying, and structuring observability events in a meaningful way.

This is where our event-driven observability framework plays a critical role.

Our framework builds the structured observability layer using EventBridge and Lambda, enabling real-time event classification, enrichment, and persistence. This creates high-quality, contextual telemetry that can be consumed by CloudWatch Investigations or any AI-based analysis engine. Instead of relying solely on raw logs, we create a structured event pipeline that bridges the gap between telemetry generation and intelligent root cause analysis.

This combination enables a complete observability stack: structured event capture, automated classification, and AI-assisted investigation.

Why You Should Build This

CloudWatch and Bedrock are powerful on their own, but the real magic happens when you connect them. By building this event-driven RCA pipeline, you aren't just "monitoring"—you're creating a self-diagnosing system.
It motivates you to trust your logs. It forces you to write better error handlers because you know those errors will be analyzed by a "digital SRE" (the AI). More importantly, it gives you back your time.
Instead of reacting to incidents, you move toward a system that actively assists in diagnosing them. Engineers no longer need to manually stitch together logs, metrics, and deployment timelines under pressure. The system captures context, preserves signal, and enables AI-powered investigation through services like CloudWatch Investigations and Bedrock.
Over time, this fundamentally changes how teams operate. Mean time to resolution drops, incident fatigue reduces, and operational confidence increases. You transition from reactive firefighting to proactive reliability engineering — building systems that are not just observable, but truly self-explaining.

If you want to see the code or deploy this yourself, check out the full repository on GitHub. Stop hunting for logs and start reading solutions.

That's Part 2.

You started this series wanting faster incident response. You now have a complete system that:

Detects failures automatically
Investigates them without human effort
Analyzes root causes with AI
Delivers actionable recommendations
All in under few minutes
Costs less than coffee 🍵

The hard part isn't the technology. It's committing to structured logging in your application. But once you do, everything else flows from that.

What's Next: RAG for Incident Memory

Instead of analyzing each incident in isolation, imagine if your system could say: "This looks like the DynamoDB timeout from 3 weeks ago. Last time, we switched to on-demand billing and it fixed it in 2 minutes." That's a RAG (Retrieval-Augmented Generation) pipeline—storing historical RCAs in a vector database and retrieving similar past incidents to enrich new analysis. Every incident becomes a learning opportunity

Resources

Feedbacks are welcome
I would love to know your thoughts and suggestions.

Lets Build

Building Event-Driven Observability on AWS Serverless (Part 1)

Sourabh Choubey — Sun, 01 Feb 2026 20:45:09 +0000

The Problem We're Solving

It's 2 AM on a Saturday. You got your teams call.

Your order service is failing silently. Customers can't check out. Revenue is bleeding out while you scramble.
You grab your laptop, login to AWS console, and start digging through CloudWatch logs. What went wrong? Was it validation? Payment? Database? Something else? You don't know yet. Thirty minutes later, you might have a guess.

This article is about never being in that situation again.

This guide walks you through building a production-grade serverless observability pipeline that triggers automated RCA.

We use SST v3 (TypeScript) for IaC and AWS managed services (CloudWatch EMF, Composite Alarms, EventBridge, Step Functions, SNS, and a small Logs Insights Lambda).

Part 1 focuses on building the demo app (Order Service), structured logging, alarms, composite alarm, EventBridge rule on the default bus, Step Functions orchestration, a FetchLogs Lambda, and email notifications via SNS.

Instead of manual log hunting, we're building a system that automatically detects failures, investigates them, and emails you the root cause—all within 90 seconds. No guesswork. Just facts.

What We're Building

An event-driven Root Cause Analysis (RCA) pipeline that works like this:

Order service fails
    ↓ (error is logged with classification)
CloudWatch detects the spike
    ↓
Alarm triggers
    ↓
EventBridge automatically invokes Step Functions
    ↓
Lambda queries historical logs for that error type
    ↓
Results emailed to your team
    ↓
Total time: 90 seconds

Before this system: MTTR = 40 minutes (manual investigation)

After this system: MTTR = 2 minutes (automated investigation)

Think of it like this. Your application is a patient. When it gets sick (fails), we don't wait for a doctor to manually examine it. We automatically run diagnostics, collect test results, and email the doctor everything they need to know.

The Scenario

Let's check any typical e-commerce platform. When a customer submits an order:

Request arrives
    ↓
Validation check (could fail → VALIDATION error)
    ↓
Payment processing (could fail → PAYMENT error)
    ↓
Save to DynamoDB (could fail → DATABASE error)
    ↓
Publish to event bus (could fail → DEPENDENCY error)
    ↓
Response sent or error returned

Each step can fail in different ways. Different failures need different fixes. That's why we classify every error—not just "it failed," but "it failed at validation" or "it failed at payment."
We'll implement a small, realistic 'Order Service' that receives API requests, writes orders to DynamoDB, emits structured logs (EMF), and publishes business events. Errors increment EMF metrics with an ErrorType dimension (VALIDATION, BUSINESS, DEPENDENCY, INFRA, UNKNOWN). We intentionally add a controlled chaos injector to produce failures for demo purposes.

Architecture Overview

High level architecture:

API Gateway -> OrderService Lambda -> DynamoDB and EventBridge
Structured logs and EMF metrics emitted by Lambda (ErrorType dimension).
CloudWatch per-ErrorType MetricAlarms -> CompositeAlarm (OR logic).
Composite alarm emits to EventBridge default bus.
EventBridge rule matches composite alarm -> Step Functions state machine (RCA pipeline).
Step Functions calls a focused FetchLogs Lambda that runs CloudWatch Logs Insights and returns recent error logs.
Step Functions publishes a curated summary to an SNS topic for email delivery.
(Part 2) The same pipeline feeds an AI analysis step for automated RCA.

Is This Framework Actually Good?

I'll be honest. Let me rate this against what you'd expect from a production system.

What Works Really Well ✅

Automatic detection: No polling, no cron jobs. When an error metric crosses the threshold, alarms fire immediately.
Structured investigation: We don't just send you a generic alert. We fetch logs from the exact time window when the error occurred, with full context (request IDs, error codes, everything).
Event-driven: The entire pipeline is asynchronous. Nothing blocks. Your API returns fast even when alarms are firing.
Cost-effective: You pay for Lambda when it runs, SNS when it sends emails, Logs Insights when you query. Nothing idle.
Observable: All of this is infrastructure-as-code (SST), so you can see exactly what's running, version control it, audit it.

What Could Be Better ⚠️

Deduplication: If 1000 orders fail, you might get 1000 RCA invocations. (Fixable with SQS dead-letter queues, but not in this initial version.)
Remediation: This system detects and reports failures. It doesn't auto-fix them. That's intentional—you probably want human judgment before auto-rolling back payments.
Multi-region: Single region only. Global platforms would need replication.
Slack/PagerDuty integration: We're using SNS email by default. Adding Slack or PagerDuty is 30 minutes of work if you need it.

The Real Value Proposition

Most teams have random log statements scattered everywhere. This framework enforces a standard: every error gets classified (VALIDATION, PAYMENT, etc.), tagged with context (requestId, userId, service), and published as a metric. Then when things break, you don't manually hunt through logs. You get a structured investigation automatically.
This framework setup focuses on this principle. Using of centralized accepted logging framework. Sending Embedded metric as part of logs and based on which we have alarms configured. Workflows gets invoked based on alarm statuses.

Not perfect, but it solves the critical problem: MTTR just dropped from 40 minutes to 2 minutes.

How It Works in simple english.

Layer 1: Detection

Your Lambda function processes an order. Something goes wrong. It logs the error with structure:

if (!userId) {
  throw new AppError(
    "User ID missing",
    "VALIDATION",  // ← This classification is key
    "MISSING_USER_ID"
  );
}

logError(error);  // Logs + emits metric automatically

Two things happen:

A structured log entry: {timestamp, level, message, errorType, errorCode, requestId, userId}
A metric to CloudWatch: OrderFailureCount with ErrorType=VALIDATION This format of adding metric makes this solution scalable where the logger utility can be reused and based on error a custom metric will be published in cloudwatch.

Layer 2: Alarming

CloudWatch watches that metric continuously. When it exceeds the threshold (default: 1 error), the alarm transitions to ALARM state.

We create one alarm per error type:

order-failure-validation-dev
order-failure-payment-dev
order-failure-database-dev
... etc

Plus a composite alarm: "ANY of these alarms is in ALARM state"

Layer 3: Triggering

An EventBridge rule listens for this specific pattern:

source = aws.cloudwatch
AND detail-type = CloudWatch Alarm State Change
AND alarm state = ALARM
AND alarm name matches order-failure-any-*

When it matches, EventBridge automatically triggers a Step Functions execution. No manual intervention. No waiting.

Layer 4: Investigation

Step Functions is your orchestrator. It calls a Lambda function that queries CloudWatch Logs Insights:

fields @timestamp, @message, requestId, errorType
| filter @message like /ERROR/
| filter errorType == "VALIDATION"
| sort @timestamp desc
| limit 20

This fetches the last 20 errors from the past 5 minutes (configurable), exactly from the time window when the alarm triggered.

Layer 5: Notification

Step Functions publishes results to SNS, which emails your team with:

Alarm name and trigger time
Error logs (timestamps, messages, request IDs)
Time window investigated

Total latency: ~90 seconds

Voila! you have received the email. I know its not the prettiest email. We are using email-json format so that if we want to ingest this somewhere else it would be easy. Also since we went with functionless approach i.e there is no lambda in between to send email, directly via SNS this minimal setup should work. On Production we can have a lambda function hooked to prettify the email, use custom email service to send email. The possibility are endless.

As you can see we are getting the logs and relevant details in JSON format that could be reused. More details on message content is added in later part of the article.

Building It: Step by Step

I would recommend to refer the attached GitHub link for more information.

Step 1: The Order Handler (Error Classification)

This is the foundation. When your Lambda processes orders, classify failures:
Below is sample code stating what one could adopt to capture essential metrics.

// services/order/handler.ts
export const handler: APIGatewayProxyHandlerV2 = async (event) => {
  const requestId =
  event?.requestContext?.requestId ??
  event?.headers?.["x-amzn-requestid"] ??
  event?.headers?.["x-request-id"] ??
  crypto.randomUUID();
  const traceId = process.env._X_AMZN_TRACE_ID;

  appendContext({
    service: "order-service",
    operation: "CreateOrder",
    requestId,
    traceId
  });

  try {
    logInfo("Create order request received", { event });
    if (!event.body) {
      throw new AppError(
        "Request body missing",
        "VALIDATION",
        "BODY_MISSING"
      );
    }

    const { userId, amount, currency } = JSON.parse(event.body);

    if (!userId || !amount || !currency) {
      throw new AppError(
        "Invalid payload",
        "VALIDATION",
        "INVALID_PAYLOAD"
      );
    }

    const orderId = `ord_${uuid()}`;
    appendContext({ orderId });

    await ddb.send(
      new PutItemCommand({
        TableName: process.env.ORDERS_TABLE!,
        Item: {
          pk: { S: `ORDER#${orderId}` },
          sk: { S: "METADATA" },
          orderId: { S: orderId },
          userId: { S: userId },
          amount: { N: amount.toString() },
          currency: { S: currency },
          status: { S: "CREATED" },
          requestId: { S: requestId },
          traceId: { S: traceId ?? "NA" },
          createdAt: { S: new Date().toISOString() }
        }
      })
    );

    await eb.send(
      new PutEventsCommand({
        Entries: [
          {
            Source: "demo.orders",
            DetailType: "OrderCreated",
            EventBusName: process.env.EVENT_BUS_NAME,
            Detail: JSON.stringify({
              orderId,
              userId,
              amount,
              currency,
              requestId,
              traceId
            })
          }
        ]
      })
    );

    logInfo("Order created", { eventType: "OrderCreated" });

    return {
      statusCode: 202,
      body: JSON.stringify({ orderId, status: "CREATED" })
    };
  } catch (err) {
    logError(err, { eventType: "OrderCreateFailed" });
    return {
      statusCode: 400,
      body: JSON.stringify({ message: "Failed to create order" })
    };
  }
};

When logError() is called, CloudWatch gets:

{
    "level": "ERROR",
    "message": "Request body missing",
    "timestamp": "2026-02-01T19:58:29.655Z",
    "service": "order-service",
    "sampling_rate": 0,
    "xray_trace_id": "1-697fb065-3f7483d74258544027bd336e",
    "operation": "CreateOrder",
    "requestId": "2bc6aa1e-cf56-4e12-a2fe-9f283466fd89",
    "traceId": "Root=1-697fb065-3f7483d74258544027bd336e;Parent=5b855ffe9a203f8f;Sampled=0;Lineage=1:141a50ec:0",
    "errorType": "VALIDATION",
    "errorCode": "BODY_MISSING",
    "retryable": false,
    "meta": {
        "eventType": "OrderCreateFailed"
    }
}

Note: Using lambda-powertools we do get important traceability out of the box. Things like requestId, traceId do help to get end-to-end picture of exactly what happened that let to this error. I recommend to go through official powertools documentation.

And simultaneously, EMF metric gets emitted:

{
    "_aws": {
        "Timestamp": 1769975909655,
        "CloudWatchMetrics": [
            {
                "Namespace": "OrderService",
                "Dimensions": [
                    [
                        "Service",
                        "Stage",
                        "ErrorType"
                    ]
                ],
                "Metrics": [
                    {
                        "Name": "OrderFailureCount",
                        "Unit": "Count"
                    }
                ]
            }
        ]
    },
    "Service": "order-service",
    "Stage": "sourabhTest",
    "ErrorType": "VALIDATION",
    "ErrorCode": "BODY_MISSING",
    "OrderFailureCount": 1
}

Why both? Logs give context. Metrics trigger alarms. Together, you get detectability + context.

Step 2: Creating Alarms (One Per Error Type)

Using SSTv3, define alarms in code:
Below thresholds are very tight. I intentionally kept it that way for testing. For Productions we can adjust based on business SLO.

// sst.config.ts
const errorTypes = ["VALIDATION", "PAYMENT", "DATABASE", "DEPENDENCY"];

const alarms = errorTypes.map((errorType) =>
  new aws.cloudwatch.MetricAlarm(`OrderFailure-${errorType}`, {
    name: `order-failure-${errorType.toLowerCase()}-${stage}`,
    namespace: "OrderService",
    metricName: "OrderFailureCount",
    statistic: "Sum",
    period: 60,              // Check every minute
    evaluationPeriods: 1,    // Trigger immediately
    threshold: 1,            // Alarm on 1+ errors (use higher in prod)
    comparisonOperator: "GreaterThanOrEqualToThreshold",
    dimensions: {
      Service: "order-service",
      Stage: stage,
      ErrorType: errorType,  // ← Must match metric dimension
    },
  }),
);

// Composite alarm: "OR" all of them
const compositeRule = pulumi
  .all(alarms.map(a => a.name))
  .apply((names) => names.map(n => `ALARM("${n}")`).join(" OR "));

const composite = new aws.cloudwatch.CompositeAlarm(
  "OrderFailureComposite",
  {
    alarmName: `order-failure-any-${stage}`,
    alarmDescription: "Any order failure detected",
    alarmRule: compositeRule,
  },
);

Why per-error-type alarms? Because you want to know what broke. Different errors need different fixes. The composite lets you trigger RCA on any error, but you still have granularity.

Note: In Part 2, we'll add Amazon Bedrock for AI-powered analysis. The IAM permissions are already in place. More on this in my next post.

// Step Functions gets Bedrock permissions
const rcaPolicy = new aws.iam.RolePolicy("OrderRcaPolicy", {
  role: rcaRole.name,
  policy: rcaStateMachine.arn.apply((arn) =>
    aws.iam.getPolicyDocument({
      statements: [
        {
          actions: ["bedrock:InvokeModel"],
          resources: [
            `arn:aws:bedrock:${region}::foundation-model/amazon.nova-premier-v1:0`,
          ],
        },
        // ... other permissions
      ],
    }).then((doc) => doc.json)
  ),
});

The Metric added creates a custom namespaces with the dimensions defined by metric. Below images shows metric data being updated based on errors observed in the system.

Step 3: Step Functions Orchestration

When the composite alarm triggers, Step Functions is invoked automatically. Here's the complete workflow for the Orchestration:

{
  "StartAt": "FetchLogs",
  "States": {
    "FetchLogs": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "${FETCH_LOGS_LAMBDA_ARN}",
        "Payload.$": "$"
      },
      "Catch": [{"ErrorEquals": ["States.ALL"], "Next": "SendFailureEmail"}],
      "Next": "SendSuccessEmail"
    },
    "SendSuccessEmail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "${ALARM_EMAIL_TOPIC_ARN}",
        "Subject.$": "States.Format('Order Failure: {}', $.detail.alarmName)",
        "Message.$": "States.JsonToString($)"
      },
      "End": true
    },
    "SendFailureEmail": {
      "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": {
        "TopicArn": "${ALARM_EMAIL_TOPIC_ARN}",
        "Subject": "RCA Failed",
        "Message.$": "States.JsonToString($)"
      },
      "End": true
    }
  }
}

Flow:

Fetch logs from CloudWatch Logs Insights
If successful → Send email with results
If failed → Send email saying RCA failed (at least you know something broke)

In Part 2, we enhance this with a parallel state that sends immediate alerts while Amazon Bedrock Nova Premier analyzes the logs with AI. You get two emails: raw data (10 sec) + AI analysis (30 sec).

Step 4: The Investigation Lambda

This Lambda is where the magic happens. It queries logs for the specific error.
It extracts error type from the alarm name, then queries CloudWatch Logs Insights for all errors matching that type within the past 10 minutes (the exact window when the alarm triggered). Since Logs Insights queries are asynchronous, it polls for results with exponential backoff, then returns structured data including timestamps, error messages, request IDs, and trace IDs—everything your team needs to start investigating.

What happens:

Extracts error type from alarm name
Queries logs for that specific error in the past 5 minutes
Polls Logs Insights until results are ready
Returns structured data: what failed, when, how many times, with full context

Testing It

Let's verify the whole pipeline works.

Test 1: Trigger an Error

We will trigger error by hitting the deployed endpoint with invalid payload.

# Send invalid request
curl -X POST "$API_URL" \
  -H "Content-Type: application/json" \
  -d ''  # Empty body triggers VALIDATION error

Based on this we do receive email with complete logs. This could then be used to do end to end check. Since we are using lambda powertools, the requestId can be used to check complete trace of request that caused the error.

Test 2: Watch the Logs

In CloudWatch Logs, we'll see the structured error. In CloudWatch Metrics, you'll see OrderFailureCount increment.

Test 3: Check Alarm State

Wait 60 seconds, then:

 aws cloudwatch describe-alarms   --query 'MetricAlarms[?Namespace==`OrderService`].[AlarmName,StateValue]'   --output table --region us-east-1


---------------------------------------------------
|                 DescribeAlarms                  |
+----------------------------------------+--------+
|  order-failure-business-sourabhTest    |  OK    |
|  order-failure-dependency-sourabhTest  |  OK    |
|  order-failure-infra-sourabhTest       |  OK    |
|  order-failure-unknown-sourabhTest     |  OK    |
|  order-failure-validation-sourabhTest  |  ALARM |
+----------------------------------------+--------+

Test 4: Verify Step Functions Executed

Post invoking endpoint with invalid details we will check the step function execution.

Test 5: Received Email

Within 90 seconds, you should receive an email with the full context. Here's what the actual email looks like:

From: AWS Notifications
Subject: Order Service Alarm: order-failure-any-dev

The email body contains a complete JSON payload with:

1. Alarm Context - What triggered:

Alarm name and transition time
Previous state (OK → ALARM)
Triggering alarm details (which specific error type caused it)

2. RCA Object - The investigation results:

"rca": {
  "Payload": {
    "alarmName": "order-failure-any-dev",
    "errorType": "VALIDATION",
    "timeWindow": {
      "from": 1769927917,
      "to": 1769928217,
      "fromReadable": "2026-02-01T06:38:37.000Z",
      "toReadable": "2026-02-01T06:43:37.000Z"
    },
    "count": 1,
    "logs": [
      {
        "@timestamp": "2026-02-01 06:42:36.945",
        "@message": "{...full structured log...}",
        "requestId": "1f00615f-7154-4964-9c14-db512efea5f8",
        "traceId": "Root=1-697ef5dc-08ed5ced0bda9d290e9aea6c",
        "errorType": "VALIDATION"
      }
    ]
  }
}

What we Get:

Time window: Exact 5-minute window that was investigated.
Error count: How many errors were found
Full logs: Each log entry with:
- Timestamp (when it happened)
- Complete error message with all context
- Request ID (for distributed tracing)
- X-Ray trace ID (for deep diving)
- Error type and code
- Order ID (for customer impact assessment)

The @message field contains the full structured log in JSON format, including:

Error classification (VALIDATION, PAYMENT, etc.)
Error code (INVALID_PAYLOAD, MISSING_USER_ID, etc.)
Whether it's retryable
Service metadata
Custom fields you've added

That's the win. Error detected → Investigated → Full context emailed. All automatic. 90 seconds.

Real-world example (redacted for privacy):
See the complete sample in the repo at examples/sample-rca-email.json showing an actual VALIDATION error with full CloudWatch Logs Insights results.

What to Think About for Production

1. Alarm Thresholds

One error = alarm. Good for testing. In production, maybe 10 errors in a minute? Depends on your traffic. Know your baseline.

2. Notifications

Email works. For production, you probably want Slack or PagerDuty. SNS integrates with both (30-minute addition).

3. Rate Limiting

If thousands of orders fail at once, you'll get thousands of RCA invocations. Add deduplication or batching to prevent Lambda/Logs Insights thrashing.

4. Cost

Estimated monthly cost at scale: $5-15

Logs Insights queries: ~$0.005 per GB scanned
Step Functions: ~$0.000025 per state transition
SNS: ~$0.50 per million emails
Lambda: Pay-per-execution

Cheap. Way cheaper than on-call engineers hunting logs.

5. Dashboard

Add a CloudWatch Dashboard to visualize:

Failure rate trends
Which error types are most common
RCA success rate

One command, huge value.

What's Coming in Part 2

Right now you get logs. In Part 2, you'll get analysis.

We'll send those logs to Amazon Bedrock , which will:

Summarize the incident in plain English
Identify likely root causes
Recommend fixes
Compare to past similar incidents

Same architecture—just add one more step that calls Bedrock and processes the response.

The Key Insight

This whole system rests on one principle: automation needs structured data.

If your logs are "oops something broke," you can't automate anything. But if your logs are structured—error type, error code, request ID, user context—suddenly you can:

Create meaningful alarms
Investigate automatically
Share context instantly
Let AI analyze patterns

The hard part isn't the AWS services (they're all standard). The hard part is getting your application to emit good logs in the first place.

Start there. Classify your failures. Emit structured data. Everything else flows from that.

References

Code

GitHub Repository:

https://github.com/Wizard-Z/aws-serverless-rca

Key Files:

sst.config.ts - Infrastructure as Code
services/order/handler.ts - Order service
services/rca/fetchlogs.ts - Investigation Lambda
shared/logging/ - Logging utilities
infra/stepfunctions/orders-rca.asl.json - RCA workflow

Part 2 coming soon. Follow me on LinkedIn to get notified.

AWS Community

Lets build!