Forem: Mursal Furqan Kumbhar

VS Code Extensions That Code For You

Mursal Furqan Kumbhar — Tue, 14 Apr 2026 10:35:13 +0000

Hello Devs 👋

If your VSCode setup still looks like it did before the AI boom, you may be coding with a bicycle in a Formula 1 race. 🏎️ Modern AI extensions are no longer novelty gadgets, they’re force multipliers. From ghost-writing boilerplate to understanding sprawling codebases and debugging at 2 AM when your brain has blue-screened, these tools can turn your editor into a caffeinated co-pilot. Here are 6 of the best AI extensions helping developers ship faster, smarter, and with fewer “why is this null?” moments.

Why does this matter?

Your editor should work as hard as you do. Most devs use VSCode, but only 10% use AI extensions. In 2026, that gap is the difference between a slow dev and a 10x dev.

1. Continue by Continue.dev

Open-Source AI Assistant. Plug in any model, like GPT-4, Claude, Llama or Mistral directly into VSCode. It:

Uses your own API Key (GPT/Claude/etc.)
Run local models via Llama
Custom slash commands & context
Full-Chat + Inline edit supports Rating: 5/5 Cost: Free | Open-Source

2. Supermaven by Supermaven Inc.

Blazing fast AI autocomplete with a 1 million context window. Its usage feels almost instant, doesn't lag and doesn't make you wait.

1,000,000 Token context.
Sees your entire repository and codebase
Sub-100ms suggestions. Truly feels instant
Understands cross-file dependencies
Free tier available Rating: 5/5 Cost: Free Plan | Pro is $10/Month

3. Pieces for Devs by Pieces.app

Your AI-Powered developer memory. It saves code snippets, context and links, and retrieves them as and when needed. It also:

Auto-saves snippets with AI Tags
Long-term memory across projects
AI Explains saved snippets on demand
Works offline with on-device AI Rating: 5/5 Cost: Free for Personal Use

4. Codeium by Exafunction

The best free Copilot alternative. It's fast, accurate and supports 70+ languages. Codeium facilitates its users in many ways, such as:

It's completely free for individual devs
Provides in-editor AI Chat for debugging
Autocompletes across 70+ programming languages
Also comes with Codeium Search and Codebase Q&A Rating: 5/5 Cost: 100% FREE | No Limits

5. Tabine by Tabnine

AI Code Completor, trained on your own codebase. This is best for teams that can't share code with external servers (mainly proprietary developers). Its main features include, but are not limited to:

Runs locally, your code stays private
Learns your codebase patterns
Works offline, no internet needed
Team model training on your repository Rating: 5/5 Cost: Pro is $12/Month

6. GitHub Copilot by GitHub (Microsoft)

The OG AI Pair Programmer. This extension auto-completes entire functions, writes tests, and explains code, all inside VSCode. Its main features include:

Inline code suggestions as you type
Copilot Chat, ask anything about your code
Multi-file context awareness
Generate unit tests automatically Rating: 5/5 Cost: $10/Month | Free for Students

Conclucion

The best developers in 2026 are not just writing code, they’re orchestrating tools. Whether you want lightning-fast autocomplete, local privacy-first AI, or a full-blown coding sidekick living in your editor, there’s an extension here for every workflow. Try a few, mix and match, and build the AI-powered setup that makes your keyboard feel slightly unfair. ⚡

Giving AI Agents “Live Memory” with Aurora Zero-ETL and Redshift Vector Search

Mursal Furqan Kumbhar — Wed, 25 Mar 2026 05:38:18 +0000

Hey everyone! Honestly, it feels so good to be sitting down and writing to you all again. I want to start with a massive apology for the radio silence and the long gap between my last few posts. Life has a funny way of getting busy, but I’ve missed this community and our deep dives into the future of tech.

We’re talking about a problem that has been bugging me all through 2025 and into 2026: AI situational awareness. Let’s dive in.

The "Goldfish" Problem: Why Your AI is Gaslighting Your Users 🐠🌀

Look, we need to have a genuine heart-to-heart about your AI agents. We’re well into 2026, and yet, most AI “agents” still have the situational awareness of a goldfish in a blender. It’s a bit heartbreaking, isn't it?

You know the feeling: You’ve spent weeks perfecting your system prompts, the UI is looking like a work of art, and your LLM is sharp. But the second a user actually does something—like upgrading a subscription, changing a delivery address, or updating their profile—the AI is the absolute last one to know. It’s like the “memory” of your agent is stuck in a permanent time lag, trapped in a 2024 batch-processing nightmare.

Imagine you’re building the ultimate AI concierge for a luxury travel app.

2:00:00 PM: Your user, Sarah, realizes she’s over the cold and cancels her flight to Paris because she’d rather sip espresso in Rome. This change hits your Amazon Aurora database instantly. Success!
2:00:10 PM: Feeling productive, Sarah asks the AI Agent, “Hey, what time does my flight leave?”
The Disaster: Because your old-school data sync (ETL) only runs every 30 minutes, the AI looks at the stale warehouse data and says, “Your flight to Paris departs at 6 PM! Don't forget your beret!” 🥖

Sarah is now officially being gaslit by an algorithm. 🤦‍♂️ The immersion? Totally gone. The trust? Shattered. This happens because the "Transactional Brain" (Aurora) isn't talking to the "Analytical Memory" (Redshift) fast enough. We need a way to move that data so quickly it feels like magic. Today, we are fixing that with Live Memory using Amazon Aurora Zero-ETL and Redshift Vector Search.

Step 1: The “Wormhole” (Zero-ETL) 🚀

In the "old days" (way back in 2024), moving data from Aurora to Redshift meant building a "Glue" pipeline. It was like trying to build a bridge between two islands using nothing but popsicle sticks, duct tape, and a whole lot of hope. It broke constantly, it cost a fortune, and it was painfully slow.

Zero-ETL is the modern way to handle this. Think of it as a native “wormhole” between Aurora and Redshift. When you enable it, AWS handles all the heavy lifting and plumbing behind the scenes. No Python scripts to debug, no Lambda triggers to monitor, and absolutely zero drama.

Why this is the “Secret Sauce” for your Agent:

Near-Zero Latency: We’re talking seconds, not minutes. By the time Sarah finishes typing her question, the data is already there.
Auto-Scaling: As your app goes viral, the teleporter grows with it.
Total Peace of Mind: You can finally stop monitoring failed Glue jobs at 3 AM. If it’s in Aurora, it’s in Redshift.

Step 2: From “Rows” to “Recall” (Vector Search) 🧠

Okay, so the data has successfully teleported into Redshift. But here’s the catch: Redshift is a warehouse full of structured tables, and AI Agents don’t really "think" in tables—they think in Vectors (mathematical representations of meaning).

In the past, you’d have to ship this data again to a dedicated vector database. But that just adds more lag! Instead, we use Redshift’s Native Vector Search. This keeps your “Live Memory” in one single, high-performance place. To make this work, we set up a Materialized View in Redshift that automatically calls Amazon Bedrock to turn those new rows into vectors on the fly.

-- The "I don't have a goldfish memory anymore" Query
CREATE MATERIALIZED VIEW live_user_context AS
SELECT 
    user_id, 
    event_description,
    -- This calls Bedrock to create a vector on the fly!
    amazon_bedrock_embed(event_description) as semantic_vector
FROM aurora_synced_data.user_logs;

Step 3: The “Aha!” Moment 🔍

Now, let's replay the Sarah scenario. When she asks about her flight, the AI Agent sends a vector query to Redshift. Redshift looks at the most recent events—including the one that landed just 10 seconds ago—and finds the “Canceled Paris / Booked Rome” entry.

The Agent responds: “I see you switched to the Rome trip—great choice! That flight leaves at 8 PM. Should I book a car to the airport?”

Boom. You just saved the user experience. 🥳 You've moved from a "chatbot" to a genuine "agent" that understands the present moment.

By The Numbers: Is it worth your time? 📈

We ran some benchmarks to see just how much of a difference this "Live Memory" architecture actually makes. When you compare the old "Batch" world to the new "Zero-ETL" world, the latency gap is massive.

Average “Brain Lag” (Latency):
- Standard S3/Glue Sync: ~480 seconds
- Zero-ETL + Redshift: ~12 seconds (A 97% improvement!)
Developer Happiness:
- Measured in “How many times I had to fix a pipeline this week”: Dropped from an average of 4 down to 0.

Pro-Tips for the Modern ML Engineer 🛠️

Keep it Serverless: Use Redshift Serverless. It scales down to zero when your agents are sleeping.
Hybrid Search is King: Don’t rely only on vectors. Combine them with regular SQL filters (like WHERE user_id = '123') to make your search lightning fast.
Capture the Vibe: Remember, this isn’t just for logs. If a user tells the AI, “I love my better half a lot,” ❤️ that sentiment gets synced and embedded instantly. The next time they ask for anniversary ideas, the agent immediately knows the priority is “High Romance.”

Final Thoughts

Anyway, it feels amazing to be back in the flow and sharing these breakthroughs with you all again. The leap from “static” AI to “live” agents is one of the most exciting shifts we’re seeing this year, and I can’t wait to see how you all implement these “wormholes” in your own stacks. If you run into any snags or just want to geek out over vector search optimizations, drop a comment below or find me over on Dev.to! I promise the next article won’t take nearly as long to reach your screens. Until next time — stay curious, keep building, and let’s make those agents a whole lot smarter. 🚀✨

RIP 12-Digit IDs: The AWS Matrix Has Finally Been Decoded

Mursal Furqan Kumbhar — Mon, 09 Feb 2026 09:41:12 +0000

Y'all... pull up a chair. We need to talk about a literal miracle. 🥂

It was 2:41 AM. PagerDuty was screaming like a banshee in my ear. My eyes were blurry, my coffee was cold, and Slack was silent in that heavy, "something-is-very-wrong" way. I logged into the AWS Console to “quickly restart a service” and get back to sleep.

Five clicks later—just as I was about to hit "Delete"—my stomach dropped. My heart hit the floor.

Wrong account. I was in Production. Nothing blew up that night. I caught myself at the last millisecond. I got lucky. But for 21+ years, millions of engineers weren’t so lucky. We’ve been squinting at 12-digit account IDs like we’re trying to decode the Matrix while the world burns around us.

But the wait is over. AWS finally added the Account Name to the top bar. It’s a "high-tech" feature that only took two decades to arrive, and it is GLORIOUS.

🏆 Welcome to the "Almost Nuked Prod" Hall of Fame

For over two decades, AWS UX followed the "Extreme Stealth" philosophy. The top bar gave you the region and your username, but the actual account context? Hidden behind a click. Pure chaos. It was like driving a car where the speedometer is inside the glove box.

This led to the Three Stages of AWS Grief:

The Over-Confidence: “I definitely logged into Sandbox. I'm a pro, I don't need to check.”
The Sudden Realization: “Wait... why does this S3 bucket have 4PB of data and a 'Do Not Delete' tag? Why is this instance type an x1e.32xlarge?”
The Panic: Aggressively smashing Ctrl+W to close 42 Chrome tabs before your shaking finger accidentally clicks 'Terminate Instance'.

Why This is Low-Key Life-Changing

This isn’t just cosmetic; it’s cognitive safety. In a multi-account world, you’re juggling dozens of environments. This update assumes that engineers are human—that we get tired, stressed, and caffeine-deprived. It’s a sanity check for your career. Good UX doesn’t assume perfection; it assumes you're human.

🛠️ Hands-On: Build Your Safety Net Today

Since you're likely rocking .ipynb files in VSCode (the ultimate dev setup, IMO), you’re used to seeing your environment in the kernel picker or the status bar. Let's get that same "don't-get-fired" energy in your browser.

1. Make it Pretty with the AWS CLI

If your account name is company-billing-prod-final-v2, it’s going to get truncated. Don't click through menus like it's 2004—we're builders! Pop this into your terminal to set a punchy, unmistakable Alias:

# Give your account a name that SCREAMS at you in the top bar
aws iam create-account-alias --account-alias PROD-SENSITIVE-ZONE

Boom. Now your top bar says PROD-SENSITIVE-ZONE in clear text instead of a random string of numbers. 💅

2. The "Visual Fire Alarm" (Layered Security)

Text is great, but color-coding saves careers. Since AWS hasn't given us a "Red Header" mode for Prod yet (maybe in the year 2047?), we have to be clever:

Unified Settings: Click your Name (top right) → Settings → Visual Mode. Choose a distinct theme for your main accounts.
Browser Profiles: This is the real pro-tip. Use different Browser Profiles (Chrome/Brave/Firefox) for different environments. I keep my Prod profile themed in "Emergency Red" with a massive warning icon. If the window isn't red, it isn't Prod.

3. The "Don't Fire Me" Python Snippet

Before running any destructive boto3 code in your Jupyter Notebook, run this check. It’s the programmatic version of "looking both ways before crossing the street."

import boto3

def safety_check():
    # Grab the current identity
    sts = boto3.client('sts')
    identity = sts.get_caller_identity()
    account_id = identity['Account']

    # Put your real Prod ID here
    PROD_ID = "123456789012" 

    print("-" * 30)
    if account_id == PROD_ID:
        print("🚨 ALERT: YOU ARE IN PRODUCTION! 🚨")
        print("Step away from the Shift+Enter key unless you're sure!")
    else:
        print(f"✅ Connected to: {account_id}")
        print("Environment: Non-Prod (Safe to play, fam)")
    print("-" * 30)

safety_check()

🚀 The Future of AWS UX

While we celebrate this "giant leap for engineer-kind," we’re still looking toward the future. Maybe one day the "Delete Database" button won't be the exact same shade of blue as the "Save" button? Maybe we'll get a "High Stakes" mode that requires a physical key turn?

Until then, appreciate the label. It might just be the thing that keeps you from becoming a "Why We Had an Outage" post-mortem.

Final Thought: The most dangerous bugs don’t live in your code; they live in interfaces that assume humans never slip. AWS finally acknowledged that we’re human. Sometimes, the biggest cloud innovation... is simply knowing exactly where you are before you hit 'Enter'. ☁️

From Monoliths to Multitaskers: Building Your AWS AI Dream Team! 🚀🤖

Mursal Furqan Kumbhar — Sat, 07 Feb 2026 07:47:02 +0000

After being away from active tech ecosystem for about a month, I have decided to shed some light on the current landscape of AI agents on AWS. And let me tell you, things have gotten wild while I was gone. We aren't just talking to single chatbots anymore; we're building entire autonomous squads.

If you’ve been building with AI lately, you’ve probably hit the "Monolith Wall." You know the one: you try to get one Large Language Model (LLM) to handle your database, write your emails, troubleshoot your network, and maybe make a decent cup of coffee. It gets confused, starts hallucinating, and suddenly your "smart" app is telling users that the server is down because it's "feeling sleepy."

The 2025 vibe is different. We are moving away from "User → LLM → Response" and toward "User → Agent Network → Coordinated Action." Think of it like a D&D party: you wouldn't ask the Barbarian to pick a lock, and you wouldn't ask the Rogue to tank a dragon. You need specialists.

Today, we’re going to learn how to orchestrate a Multi-Agent System (MAS) on AWS Bedrock. Grab your coffee (or mana potion), and let’s dive in!

🛠 The Secret Sauce: What is a "Bedrock Agent"?

Before we build the team, we need to understand the individual "hero." A Bedrock Agent isn't just an LLM with a fancy name. It's a system with three core parts:

The Brain (Model Provider): Usually something like Claude 3.5 Sonnet or the zippy new Amazon Nova series.
The Manual (System Prompt): The instructions that tell the agent, "You are a world-class SRE," and give it boundaries (like "Don't delete production, please").
The Toolbelt (Action Groups): These are the superpowers! This is where the agent calls AWS Lambda functions to actually do things in the real world.

🎭 The Multi-Agent Party: Meet the Supervisor

Why have one agent when you can have five? In a Supervisor-Agent Pattern, you have one "Dungeon Master" (the Supervisor) who listens to the user and decides which specialist to call.

Imagine we're building a Telco Network Operations Assistant. Our party looks like this:

Role	Specialty	Data Source
The Supervisor	Orchestration & Planning	Routes the query to the right specialist.
The Alchemist (KPI Agent)	Performance Metrics	Checks throughput and latency.
The Scout (Alarm Agent)	Real-time Status	Looks for active site failures.
The Chronicler (Log Agent)	History & Patterns	Scans CloudWatch logs for anomalies.

🏗 Hands-On Walkthrough: Building the "Site Ops Hero"

Let's get our hands dirty. We want to ask: "What's the status of site_dallas_001?" and get a correlated answer.

Step 1: The Magic Lambda (Your Action Group)
Your agent needs a bridge to the real world. We’ll use a Lambda function. When the agent calls it, it sends a JSON event that looks like this:

{
  "actionGroup": "NetworkTools",
  "apiPath": "/get-site-alarms",
  "parameters": [{"name": "site_id", "value": "site_dallas_001"}]
}

Tip: Bedrock events aren't the same as API Gateway events! You’ll need to parse the apiPath and parameters directly.

Step 2: Set up the Specialist Agents
In the Bedrock Console (or via Boto3), create your specialist agents. Give them Knowledge Bases so they can read your documentation stored in S3.

Pro-tip: Use Titan Text Embeddings v2 to turn your PDFs into "vector gold" that the agent can search in milliseconds.

Step 3: Summon the Supervisor
Create a new agent and enable Collaboration Mode. Associate your specialist agents as "collaborators." Now, when you talk to the Supervisor, it creates a plan:

Ask the Scout for active alarms.
Ask the Alchemist if performance is dropping.
Combine the answers and tell the human exactly what’s broken.

💰 Budgeting Your "Mana" (Cost Optimization)

Running a whole team of AI agents can get pricey if you're not careful. Here’s how to keep your AWS bill from going "Critical Hit" on your wallet:

Tiered Intelligence: Use the "big brain" (Claude 3.5 Sonnet or Nova Premier) for the Supervisor who needs to plan. Use the "fast brain" (Nova Lite or Micro) for simple sub-tasks.
Prompt Caching: If you’re sending the same 50-page manual to your agent every time, Prompt Caching can save you up to $90\%$ on input tokens.
Batch Inference: If you’re doing something that doesn’t need an answer right now (like summarizing yesterday's logs), use Batch Mode for a 50% discount.

🛡 Staying Safe (Guardrails)

You don't want your agent going rogue. Use Amazon Bedrock Guardrails to set "Denied Topics." For example, you can explicitly forbid the agent from giving out employee home addresses or venting about its boss. You can even set up Contextual Grounding Checks to make sure the agent only answers based on your data, not its own imagination (aka hallucinations).

🏁 Conclusion: Your Adventure Awaits!

Building multi-agent systems on AWS is like moving from playing a single-instrument solo to conducting a full philharmonic orchestra. It’s more complex, sure, but the results are magical.

Whether you're automating SRE tasks or building a Pokémon battle advisor, the tools are all there in Bedrock.

What are you waiting for? Go build something awesome! 🚀✨

P.S. When you're done, remember to delete your S3 buckets and Lambda functions so you don't wake up to a "Surprise Invoice" Boss Fight.

The AWS Lambda Setting I Wish I’d Changed Earlier

Mursal Furqan Kumbhar — Tue, 30 Dec 2025 09:10:50 +0000

I've been using AWS Lambda for quite a few years, and like many people, I left the architecture on its default settings, i.e. x86 (Without thinking 2x). It worked really well, it scaled as needed, but the truth unfolded when invoices showed up. End of Story. Fine Della Storia.

Upon closer examination of the situation, I paid more attention to the execution time & cost, and then I thought to experiment with Arm64, i.e. AWS Graviton. And the rare cloud optimisations that actually delivered, followed. 🎇 A VISIBLE WIN 🎇
.
.
.
Now the main Question

What led me to try Arm64?

The answer's as simple as it gets: A few Lambda handling API Requests & Background jobs were being called 1,000,000s of times per month and frankly, each one looked quite cheap, individually. But when bunched together, they burned a steamy hole in my pocket every month.

Now being a lazy_developer who uses snake cases while programming, I said NO to refactoring the logic or the code. I just wanted a clean performance win. And that's when... Arm64 looked like a low-risk bet to me.

Time taken towards the CHANGE!!!

Now, when I say I moved from x86 to Arm64, the first thing to pop in your head would be, "Nah! It's gonna take forever to change." Right? WRONG. I changed the lambda architecture from x86 to Arm64, redeployed and tested it within a matter of hours (And that's because I chose NOT to read the documentation first 🥲. It could have been sooner). That was it. No code was changed, no dependencies were broken, and no surprises. This alone was... unexpected.

What were my ACTUAL Observations

The moment I started moving only a few Lambdas functions to Arm64, the execution time dropped noticeably, especially for CPU-based logic. And guess what... the Cold Start felt snappier on lightweight Node.js & Python functions. In the end, the monthly lambda cost dip without touching memory allocation felt like a cherry on top.

So, nothing dramatic for every invocation, but at scale, the difference was truly impossible to ignore.

Reality Check: The Cost

Lambda billing isn't rocket science. It's simple maths:

Execution time * Memory * Invocations

If your function runs just 20-30 milliseconds faster, and it is executed millions of times, that tiny change in the execution speed turns into a new Apple Watch (Yes, no kidneys needed 😜). This was the moment Arm64 stopped feeling like an optimisation experiment & started feeling like a worthy replacement to x86 as default choice

Why did Arm64 work the best for ME?

In my small but meaningful experience, Arm64 shone brighter than star Sirius, when I used it for API Gateway-backed Lambdas, Event Driven pipelines, background jobs, schedulers, data processing & automation or even for lightweight ML inference. In short, I opted for Arm64, where I used modern serverless workloads with updated runtimes.

Where I would never use Arm64?

It's not like I am completely against x86 now. I still used it in my project for some scenarios. For example:

I kept x86 for older functions with legacy native binaries
Or for rare dependencies that weren't Arm-ready, yet
Or for the code, that I didn't want to touch, being that close to the deadline. (Programmers, IYKYK). For those who don't, we don't want the project go

What changes in the future?

In the future, if I am starting a new serverless project (Or today, who knows):

I will definitely go for Arm64, by default
I am gonna only move back to x86, only & only, if something explicitly breaks (Or my client doesn't love their 💵)
Will benchmark early, instead of assuming

Conclusion

Arm64 on AWS Lambda isn't a future feature, it's just a teeny tiny and quiet upgrade that's already paying off its cost in real-life systems. It's not theoretical. IT's PRACTICAL. Well, if you care about factors such as performance, cost & efficiency and you're running modern hsuper heavy serverless workloads, listen to me (Or read this carefully): TRY IT ONCE. 9/10 Chances? Like me, you won't turn back 😉.

Once again, this was my personal experience.

Buon Anno

HappyCoding

How I Automated Document Insights Using AWS Textract, Bedrock, and QuickSight

Mursal Furqan Kumbhar — Fri, 12 Sep 2025 17:35:28 +0000

Hello Everyone 👋

So I had a bunch of scanned invoices and PDFs lying around from an old freelance gig. Manually reading them and extracting data felt like a punishment. Instead of going through them one by one, I decided to try something smarter (I know, common Developer Syndrome 😜), to build a small AWS workflow that could:

Read scanned docs,
Summarise key info,
Convert it to structured data, and
Visualise everything inside a dashboard.

Here’s how I did it, step by step. You can follow the same process and tweak it for contracts, reports, receipts, whatever you’ve got.

✅ Step 1: Set Up Your AWS Environment (Production-ready)

1) Pick a single region that supports everything

Goal: Keep S3, Textract, Bedrock, and QuickSight in the same region to avoid cross-region latency/fees.
How to choose: Check that your chosen region offers Bedrock (model availability varies) and Textract. If QuickSight isn’t in that region, use Athena + Glue as a bridge (S3 anywhere → Athena in QuickSight’s region).
Edge cases
- Bedrock not in your favorite region: run Textract + S3 where your data lives, then send plain text only to Bedrock in a supported region (privacy + cost trade-off).
- Corporate constraint on regions: document the exception and add an S3 Replication rule or use Athena.

2) Create an S3 bucket with the right defaults

Name: doc-insights-<env>-<region>-<account-id> (e.g., doc-insights-prod-eu-central-1-123456789012)
Settings to enable right away
- Block Public Access: ON (all four toggles).
- Versioning: ON (lets you recover bad parses).
- Default encryption: SSE-KMS (customer-managed key) for auditability.
- Object Ownership: Bucket owner enforced (disables ACLs, simplifies perms).
- Lifecycle rules:
- Move raw/ to infrequent access after N days.
- Expire tmp/ after M days.
- Keep processed/ longer for audits.
Folder layout (simple + scalable)

  s3://doc-insights-bucket/
    raw/               # uploads: PDFs, images
    textract-json/     # raw JSON from Textract (per doc)
    extracted/         # normalized tables/kv (CSV/Parquet)
    summaries/         # Bedrock text summaries (JSON/MD)
    processed/         # final CSVs for BI
    logs/              # app / pipeline logs

Edge cases
- Big uploads / flaky networks: use multi-part upload (SDK/CLI handles this).
- PII/PHI compliance: restrict bucket to VPC endpoints (below), rotate KMS keys, and enable CloudTrail data events for S3.
- Cross-account access: prefer bucket policy + role assumption over sharing long-lived keys.

3) IAM roles & least-privilege policies

Create three roles (one per workload) and scope them tightly:

a) Textract role (used by your Lambda/ETL or notebook)

Trusted entity: lambda.amazonaws.com (or your compute service)
Inline policy (example)

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect":"Allow","Action":["textract:AnalyzeDocument","textract:StartDocumentAnalysis","textract:GetDocumentAnalysis"],"Resource":"*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::doc-insights-bucket"]},
    {"Effect":"Allow","Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}
  ]
}

Use Start/Get APIs for async jobs (multi-page PDFs); AnalyzeDocument is fine for small/sync runs.

b) Bedrock role (for summarization)

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["bedrock:InvokeModel","bedrock:InvokeModelWithResponseStream"],"Resource":"*"},
    {"Effect":"Allow","Action":["s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/*"]},
    {"Effect":"Allow","Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}
  ]
}

Limit Resource to the specific model ARNs once you pick them (Claude, Mistral, etc.).

c) QuickSight service role access to S3

Inside QuickSight → Admin → Security & permissions, grant access to your bucket. Backing IAM should allow:

{
  "Version":"2012-10-17",
  "Statement":[
    {"Effect":"Allow","Action":["s3:GetObject"],"Resource":["arn:aws:s3:::doc-insights-bucket/processed/*"]},
    {"Effect":"Allow","Action":["s3:ListBucket"],"Resource":["arn:aws:s3:::doc-insights-bucket"]}
  ]
}

Edge cases
- SSE-KMS blocks reads: add the QuickSight role principal to your KMS key policy.
- Cross-account S3: add a bucket policy that allows the QuickSight account role + use a manifest file.

4) KMS key policy that won’t bite you later

Add all compute roles and the QuickSight role to the key policy (not just IAM perms), e.g.:

{
  "Version":"2012-10-17",
  "Statement":[
    {"Sid":"EnableRoot","Effect":"Allow","Principal":{"AWS":"arn:aws:iam::ACCOUNT:root"},"Action":"kms:*","Resource":"*"},
    {"Sid":"AllowUseOfKey","Effect":"Allow","Principal":{"AWS":[
      "arn:aws:iam::ACCOUNT:role/TextractRole",
      "arn:aws:iam::ACCOUNT:role/BedrockRole",
      "arn:aws:iam::ACCOUNT:role/service-role/QuickSight-ROLE"
    ]},"Action":["kms:Encrypt","kms:Decrypt","kms:GenerateDataKey","kms:DescribeKey"],"Resource":"*"}
  ]
}

Edge cases
- AccessDenied (kms:Decrypt) in QuickSight/Athena: 99% of the time it’s the key policy missing the service role.

5) Networking (private, compliant, and fast)

Create VPC interface endpoints (PrivateLink) for:
- S3 (Gateway endpoint), Textract, Bedrock, STS, Logs, etc.
Lock your compute (Lambda/ECS/EC2) to no-internet and route AWS traffic via endpoints.
Edge cases
- QuickSight can’t sit in your VPC: it’s managed; keep the S3 bucket policy open to QuickSight role (not public) and allow KMS for that role.
- Corporate egress proxy: verify Bedrock/Textract endpoints are reachable or use endpoints.

6) Turn on Amazon Bedrock access (once per account/region)

In the Bedrock console, enable model access (Claude/Mistral etc.). Some models require an opt-in.
Edge cases
- Model not visible: you’re in a region without that model, or access not granted yet.
- Throughput errors: request service quota increases for InvokeModel if you batch summaries.

7) Enable Amazon QuickSight

Choose the same region as your S3 if possible.
In Security & permissions, tick your bucket and (if needed) Athena access.
Decide SPICE vs Direct Query:
- SPICE (in-memory) is faster; watch capacity limits.
- Direct is fine for small CSVs; for large data, prefer Parquet in S3 via Athena.
Edge cases
- CSV > size limit / too many rows: convert to Parquet and query via Athena.
- Timezone mismatches: set dataset time zone explicitly; normalize date formats (ISO-8601).

8) Budget, cost alerts, and audit

AWS Budgets: create a monthly budget with email alerts (S3 + Textract + Bedrock + QuickSight).
CloudTrail data events: ON for S3 (per-object audit).
CloudWatch Logs: centralize Lambda/app logs to logs/.
Edge cases
- Sudden spikes: enable S3 object-level CloudWatch metrics and add alarms on object count/size in raw/.

9) File formats & naming that make your life easy

Uploads (raw): accept PDF/JPEG/PNG/TIFF; normalize on ingest.
Outputs: prefer CSV (QuickSight-friendly) and/or Parquet (Athena-friendly).
Names: raw/{yyyy}/{mm}/{dd}/{vendor}/{fileid}.pdf processed/{yyyy}/{mm}/{dd}/invoices.csv
Edge cases
- Weird encodings: force UTF-8, escape delimiters, include header row.
- Duplicate filenames: use a UUID suffix or checksum in the key.

10) Optional: IaC and CLI quick start

Create bucket (CLI)

aws s3api create-bucket --bucket doc-insights-bucket --create-bucket-configuration LocationConstraint=$AWS_REGION
aws s3api put-bucket-encryption --bucket doc-insights-bucket --server-side-encryption-configuration '{
  "Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"arn:aws:kms:REGION:ACCOUNT:key/KEY_ID"}}]}'
aws s3api put-bucket-versioning --bucket doc-insights-bucket --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket doc-insights-bucket --public-access-block-configuration \
'{"BlockPublicAcls":true,"IgnorePublicAcls":true,"BlockPublicPolicy":true,"RestrictPublicBuckets":true}'

Example lifecycle (CLI)

aws s3api put-bucket-lifecycle-configuration --bucket doc-insights-bucket --lifecycle-configuration '{
  "Rules":[
    {"ID":"raw-to-ia","Filter":{"Prefix":"raw/"},"Status":"Enabled","Transitions":[{"Days":30,"StorageClass":"STANDARD_IA"}]},
    {"ID":"expire-tmp","Filter":{"Prefix":"tmp/"},"Status":"Enabled","Expiration":{"Days":7}}
  ]}'

📄 Step 2: Use Amazon Textract to Extract Text from Documents (Production-grade)

Textract can do three things you’ll care about:

OCR (lines/words) → DetectDocumentText
Generic forms & tables → AnalyzeDocument with FORMS, TABLES, optionally QUERIES
Invoice/receipt specialization → AnalyzeExpense (often best for invoices)

Rule of thumb

If it’s an invoice/receipt, try AnalyzeExpense first (it returns normalized fields like VENDOR_NAME, INVOICE_TOTAL).

If it’s a contract/form/report, use AnalyzeDocument with FORMS and TABLES.

Use QUERIES to target specific fields (e.g., “What is the due date?”) when layout varies wildly.

1) Sync vs Async: choose the right API

Synchronous (single call; immediate result)
- AnalyzeDocument (good for single-page or small docs; great for tests)
- DetectDocumentText (basic OCR)
Asynchronous (submit job → poll or page through results)
- StartDocumentAnalysis + GetDocumentAnalysis
- Required for multi-page and larger PDFs/images
- Supports pagination via NextToken

Guidance: Use sync for quick console/dev tests or one-pagers; use async for anything real (multi-page, batch).

2) Pre-processing (massively improves accuracy)

Input formats: PDF, PNG, JPEG, TIFF work well; keep scans clean.
Quality: Aim for readable text (good contrast, minimal blur). If possible:
- Deskew, denoise, increase contrast, convert to grayscale/PNG.
- Fix rotation (many scanners rotate by 90° or 180°).
Encrypted or password-protected PDFs: decrypt before upload.
Color: Grayscale often beats noisy full-color scans.
Multi-page splits: If you hit page/file limits, split before upload.
File names: Include vendor/date hints in the key to simplify downstream routing (e.g., raw/2025/09/11/vendorX/invoice_123.pdf).

3) Minimal but robust code paths (Python/boto3)

A. Synchronous: quick test for forms + tables

import boto3

textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_document(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}},
    FeatureTypes=["FORMS", "TABLES"]
)

blocks = resp.get("Blocks", [])
# → Save raw JSON, parse KEY_VALUE_SET/TABLE blocks later

B. Asynchronous: production for multi-page docs (handles pagination + retries)

import boto3, time, itertools

textract = boto3.client("textract", region_name="eu-central-1")

job = textract.start_document_analysis(
    DocumentLocation={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "long-invoice.pdf"}},
    FeatureTypes=["FORMS","TABLES"]
)
job_id = job["JobId"]

def get_all_pages(job_id, sleep=2, max_wait_s=900):
    start = time.time()
    pages = []
    next_token = None

    # Wait for completion (simple poll)
    status = "IN_PROGRESS"
    while status == "IN_PROGRESS":
        res = textract.get_document_analysis(JobId=job_id, NextToken=next_token) if next_token \
              else textract.get_document_analysis(JobId=job_id)
        status = res["JobStatus"]
        if status == "IN_PROGRESS":
            # If first page says in progress, wait; otherwise collect & keep paging
            if "Blocks" in res:
                pages.append(res)
                next_token = res.get("NextToken")
                if not next_token:
                    time.sleep(sleep)
            else:
                time.sleep(sleep)
            if time.time() - start > max_wait_s:
                raise TimeoutError("Textract job took too long.")
        elif status == "SUCCEEDED":
            pages.append(res)
            while res.get("NextToken"):
                res = textract.get_document_analysis(JobId=job_id, NextToken=res["NextToken"])
                pages.append(res)
        else:
            raise RuntimeError(f"Textract job failed: {res.get('StatusMessage', status)}")
    return pages

pages = get_all_pages(job_id)
# Combine blocks across pages for parsing
all_blocks = list(itertools.chain.from_iterable(p["Blocks"] for p in pages))

C. Invoices/Receipts: specialized `AnalyzeExpense` (often higher accuracy on totals/dates)

import boto3

textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_expense(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}}
)

# resp["ExpenseDocuments"][0]["SummaryFields"] includes types like:
# VENDOR_NAME, INVOICE_RECEIPT_DATE, INVOICE_RECEIPT_ID, TOTAL, DUE_DATE

D. Targeted fields with `QUERIES` (when layouts vary)

import boto3
textract = boto3.client("textract", region_name="eu-central-1")

resp = textract.analyze_document(
    Document={"S3Object": {"Bucket": "doc-insights-bucket", "Name": "invoice1.pdf"}},
    FeatureTypes=["QUERIES","TABLES"],  # you can mix with FORMS too
    QueriesConfig={
        "Queries": [
            {"Text": "What is the invoice number?", "Alias": "invoice_no"},
            {"Text": "What is the total amount?", "Alias": "total_amount"},
            {"Text": "What is the due date?", "Alias": "due_date"}
        ]
    }
)

You can run QUERIES + TABLES together to pull exact fields and still capture line items.

4) How to parse Textract blocks safely (just enough for Step 2)

Textract represents a graph of Blocks with Relationships. The key ones:

PAGE, LINE, WORD
KEY_VALUE_SET (with EntityTypes: KEY or VALUE)
TABLE → CELL (cell has row/col indices)
QUERY → ANSWER

Parsing outline (KV):

Build a dict of blocks by Id.
Collect all KEY_VALUE_SET blocks.
For each KEY block, follow Relationships to its VALUE block.
Extract text by concatenating child WORD blocks; track Confidence.
Normalize keys (vendor, invoice no, etc.) and keep the BoundingBox for traceability.

Parsing outline (TABLE):

Find TABLE blocks, get their child CELLs.
Use (RowIndex, ColumnIndex) to reconstruct rows.
Heuristically detect headers (first row or bold-ish/upper-case hints if present in text).
Output as CSV; preserve page number.

We’ll do the full CSV/JSON conversion in Step 3. Here we only ensure you’ve got clean raw JSON saved and a clear parser plan.

5) Storage & idempotency

Always save raw responses (one file per source doc):
- textract-json/<doc-id>.json (for AnalyzeDocument/DetectDocumentText)
- textract-json/<doc-id>.expense.json (for AnalyzeExpense)
Add metadata to input S3 objects (e.g., processed=true, textract_job_id=...) to avoid reprocessing.
Use S3 object ETag/size as a cheap duplicate detector.

6) Confidence, validation & fallbacks

Confidence thresholds: treat fields under a threshold (e.g., < 0.8) as “needs review.”
Numeric sanity checks: parse amounts with a currency regex, verify:
- sum(line_items) ≈ total (± a few cents, taxes/discounts considered)
- date is valid & plausible (e.g., due date ≥ invoice date)
Disambiguation: fight common OCR confusions:
- O↔0, l/I↔1, S↔5, B↔8
Fallback order for invoices:

AnalyzeExpense
AnalyzeDocument (FORMS+TABLES)
DetectDocumentText + regex heuristics (as a last resort)
- Human-in-the-loop: flag low-confidence docs to a review queue (CSV row with a needs_review=1 flag).

7) Permissions & security edge cases (most common causes of failure)

S3: role needs s3:GetObject, s3:ListBucket, and if encrypted, kms:Decrypt on the CMK.
KMS key policy: include the calling role principal (not just IAM permissions).
Cross-account: use bucket policy + role assumption; ensure KMS allows the assumable role’s principal.
Private networking: if using VPC-only compute, add VPC endpoints for Textract/S3/STS/Logs.

8) Scaling, throughput, and cost control

Batching: drive Textract from a queue (SQS) or Step Functions; cap concurrency to avoid throttling.
Retries: exponential backoff on ThrottlingException and transient network errors.
De-dup: check if a doc was already processed (S3 metadata) before calling Textract again.
Specialization saves money: AnalyzeExpense or QUERIES can reduce post-processing and re-runs.

9) Troubleshooting cheat-sheet

Symptom	Likely cause	Fast fix
`AccessDenied` on S3 or KMS	Missing `s3:GetObject` / `kms:Decrypt` or KMS key policy excludes role	Add actions to role and role principal to KMS key policy
Job stuck “IN_PROGRESS” forever	Not polling pages / ignoring `NextToken`	Implement full pagination loop until `NextToken` is empty
Empty or low-quality text	Bad scans (blur, skew), wrong rotation, low contrast	Preprocess image: deskew, increase contrast, convert to PNG, rotate correctly
Fields missing on invoices	Using generic forms instead of expense API	Switch to `AnalyzeExpense` or add `QUERIES`
Dates/amounts parsed wrong	Locale/regex issues or OCR confusions	Normalize formats (ISO-8601), add currency/number regex, apply character-swap heuristics
QuickSight can’t see outputs	Using KMS + missing QS access	Add QS role to bucket policy and KMS key policy

10) Console workflow (great for validation)

Upload a sample to S3 → Textract Console → Analyze.
Toggle Forms + Tables, confirm highlights match expectations.
For invoices, test Expense analysis and compare field names (often cleaner).
Save the result JSON and move on to parsing (Step 3).

✏️ Step 3: Convert Textract Output to Structured Data (CSV or JSON)

1) Clean & normalize the extracted data

Keep originals: store the raw Textract JSON (read-only) so you can re-parse later without re-running Textract.
Whitespace & casing: trim, collapse double spaces; keep a _raw copy for each value if you’ll audit later.
Key normalization (FORMS/QUERIES): map variants to canonical keys:
- invoice no, inv #, invoice number → invoice_number
- total, grand total, amount due → total
- supplier, vendor, billed by → vendor_name
- invoice date, date → invoice_date; due date → due_date
Dates: convert to ISO-8601 (YYYY-MM-DD). If ambiguous/parsing fails, leave blank and flag needs_review.
Numbers & currency: strip thousands separators, unify decimals (.), extract currency symbol/code ($→USD, €→EUR, £→GBP, ₨→PKR, ₹→INR). If none, set a default and flag for review.
Confidence tracking: carry Textract confidence (0–100). Anything below your threshold (e.g., 80) sets needs_review = 1.
OCR confusions: apply gentle corrections when context demands numeric values (O↔0, I/l↔1, S↔5, B↔8) but still keep needs_review if confidence was low.

2) Choose your outputs

CSV (analytics-ready):
- KV (long format): doc_id, page, key_norm, value, confidence, needs_review
- Tables (per page/table): doc_id, page, row, col, text, confidence, is_header
- Invoice summary (if using Expense API): one row per document with normalized fields
JSON (traceability): keep structured JSON when you need bounding boxes/page indices for audits.
(Optional) Parquet: for Athena/QuickSight at scale.

3) Dump key–value pairs to CSV

What to capture: each detected pair as one row in a long table:
- doc_id, page, key_raw, key_norm, value, conf_key, conf_val, needs_review
Duplicates: keep all candidates. Later, pick the highest confidence (ties → closest to the label “Total”, latest on the page, or appears in a summary box).
Missing values: if a KEY has no VALUE, record value = "" and set needs_review = 1.
Multi-line values: join lines with a single space; preserve original in value_raw if you keep audits.

4) Convert tables from JSON to CSV

Reconstruct using cells: one row per cell with indexes:
- doc_id, page, row, col, text, confidence, span_rows, span_cols, is_header
Headers: default to row 1 = header; mark is_header = 1. If row 1 has poor confidence but contains many alphabetic tokens vs numeric rows below, still treat as header.
Merged cells: repeat the merged text in the spanned area and record span_rows/span_cols for fidelity.
Multi-page tables: keep separate by page; your business logic decides whether to stitch them later.

5) Build the simple “wide” table for your dashboard

Target columns from your outline:

Invoice No (`invoice_number`)	Date (`invoice_date`)	Vendor (`vendor_name`)	Amount (`total`)	Due Date (`due_date`)
INV-123	2025-08-03	Alpha Supplies	423.90	2025-09-02

Population priority:

Expense API (if used): INVOICE_RECEIPT_ID, INVOICE_RECEIPT_DATE, VENDOR_NAME, TOTAL, DUE_DATE.
FORMS/QUERIES: after key normalization and numeric/date parsing.
TABLES/OCR fallback: totals row or summary box heuristics (e.g., last column named Amount, Total, Balance).

Also add helpful derived fields:

currency, source (expense/forms/ocr), needs_review, month, year.

6) Quality & integrity checks

Arithmetic sanity: if you have subtotal and tax, verify subtotal + tax ≈ total (small tolerance for rounding). If not, needs_review = 1.
Date logic: due_date ≥ invoice_date; otherwise needs_review = 1.
Negatives & zeros: total must be > 0 for invoices; credits/notes may be negative tag with doc_type = credit_note.
Duplicates: same invoice_number appearing across different doc_ids → keep the one with higher composite score (weighted confidence of critical fields) and mark others as superseded.
Two documents in one file: if you detect two distinct vendors or invoice numbers in one doc_id, set multi_doc_suspected = 1 and route to review.
Locale drift: month names/currency symbols outside your default locale → still parse but set locale_detected and needs_review if parsing relied on heuristics.
Handwritten/low DPI scans: values may be fragmented; prefer QUERIES for targeted fields and flag low confidence.

7) Foldering

textract-json/            # raw API outputs (never edited)
extracted/
  kv/                     # <doc_id>_kv.csv
  tables/                 # <doc_id>_table_<page>.csv
  expense/                # <doc_id>_summary.csv, <doc_id>_line_items.csv (if used)
processed/
  invoices_wide.csv       # your dashboard input (or Parquet partitioned by year/month)

🧠 Step 4: Summarise the Document Using Bedrock

1) What you’ll summarise

Feed Bedrock only what it needs:

Facts you already extracted in Step 3 (invoice_number, vendor_name, total, due_date, currency, etc.).
Top signal text: page headers, totals area, vendor/address block, and (for invoices) the line-items table flattened to a few lines (don’t paste thousands of rows).
Optional full text: chunked by page if you need narrative context.

Redact or truncate PII that isn’t needed (addresses beyond city/country, phone, full account numbers). Keep the doc_id so you can trace back.

2) Output format

Ask Bedrock for structured JSON + a short narrative. Example JSON schema:

{
  "doc_id": "string",
  "summary": "1–3 sentence human summary",
  "key_fields": {
    "vendor_name": "string|null",
    "invoice_number": "string|null",
    "invoice_date": "YYYY-MM-DD|null",
    "due_date": "YYYY-MM-DD|null",
    "currency": "string|null",
    "total_amount": "number|null"
  },
  "observations": [
    "string"  // anomalies, discounts, late fees, notes
  ],
  "data_quality": {
    "missing_fields": ["string"],
    "suspected_errors": ["string"],
    "confidence_note": "string"
  }
}

This slots straight into your processed/ layer or QuickSight as a secondary dataset.

3) Prompting

Use a system + user style prompt; instruct the model not to invent data.

System

You are a careful financial document assistant. Use ONLY the supplied text and fields.
If a field is missing or unclear, return null and add a helpful note in data_quality.
Output must be valid JSON matching the provided schema. Do not include any other text.

User

DOC_ID: {{doc_id}}

FACTS (from extraction):
- vendor_name: {{vendor_name}}
- invoice_number: {{invoice_number}}
- invoice_date: {{invoice_date}}
- due_date: {{due_date}}
- currency: {{currency}}
- total_amount: {{total}}

EXCERPTS (trimmed; page-tagged):
[PAGE 1]
{{top_of_page_text}}
[PAGE 2 - totals box]
{{totals_area_text}}
[OPTIONAL line items]
{{few_key_lines}}

TASK:
1) Produce a JSON object exactly matching this schema:
{{paste schema}}
2) Base fields on FACTS when available; only use EXCERPTS to confirm or fill blanks.
3) If numbers disagree, prefer FACTS but note the discrepancy in `data_quality.suspected_errors`.
4) Do not guess. Use `null` for unknowns.

Why this works

It anchors the model to your extracted fields (reduces hallucinations).
It forces a predictable JSON payload for downstream use.
It adds a quality channel for “known unknowns”.

4) Model choice & params

Claude (Anthropic): strong on instruction following & JSON fidelity.
Mistral: concise, fast; good when payloads are small and schema is simple.
Parameters: keep temperature low (≈0.0–0.2) for factual stability; set max tokens tight (e.g., 600–900) for cost control.

Tip: If JSON validation fails sometimes, wrap with a “fix to valid JSON” pass (or enable JSON mode where available).

5) Long documents or Map/Reduce

For multi-page or very verbose docs:

Map step: summarize each page (fixed schema, plus page-level anomalies).
Reduce step: feed all page summaries (+ your Step-3 facts) into a second prompt that merges into a single final JSON, breaking ties in favor of Step-3 facts.

This avoids token limits and preserves page context.

6) Robust invocation

Use bedrock-runtime with the model you enabled (ids vary by region/provider). Two common patterns:

Non-streaming (simple)

import json, boto3, time
brt = boto3.client("bedrock-runtime", region_name="YOUR_REGION")

def invoke_bedrock(model_id, system_prompt, user_prompt, max_tokens=800, temperature=0.1):
    body = {
        "anthropic_version": "bedrock-2023-05-31",
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_prompt}
        ],
        "max_tokens": max_tokens,
        "temperature": temperature
    }
    resp = brt.invoke_model(modelId=model_id, body=json.dumps(body))
    out = json.loads(resp["body"].read())
    # Extract the text depending on provider; for Claude:
    text = "".join([c.get("text","") for m in out.get("content",[]) for c in [m]])
    return text

Streaming

Stream when you want low latency for UI, otherwise skip (adds complexity).

Note: Exact request/response shapes differ by model family. Keep the modelId and payload format configurable and test in the Bedrock console first.

7) Validation & safety rails

JSON validation: run a quick schema check (pydantic/jsonschema). If invalid, re-prompt the model with: “Return only valid JSON conforming to this schema. Do not add commentary.”
No-invention rule: instruct “If unsure, use null and add a data_quality note.” Penalize hallucinations by rejecting outputs that add fields you didn’t request.
Discrepancy handling: when EXCERPTS disagree with FACTS, keep FACTS and add a note like: “Totals area shows 432.90 but extracted total is 423.90 (difference 9.00).”
PII minimization: don’t pass line-item text unless needed; mask account numbers (e.g., ****1234).
Cost control: truncate excerpts, avoid full dumps; cap max_tokens; batch summarize.

8) Edge cases

Edge case	What to do in prompt / output
Missing total or due date	Return `"total_amount": null` / `"due_date": null`; add a `data_quality.missing_fields` entry.
Multiple totals (invoice vs balance)	Prefer FACTS; if absent, pick the one labeled “Total” near summary; note the alternative.
Conflicting currency symbols vs codes	Prefer the currency from FACTS; if excerpt differs, note discrepancy.
Handwritten / low DPI	Expect nulls and notes; don’t interpolate from context.
Multi-currency line items	Set currency to document currency; add observation like “Line items include mixed currency not used for total.”
Multi-doc PDF	If two vendors or two invoice numbers appear, say so in `observations` and add `suspected_errors`.
Non-English docs	Add “Output in English.” Keep dates numeric (YYYY-MM-DD).
Extremely long inputs	Use Map/Reduce; only pass per-page top areas and final consolidation.

9) Example “good” output

{
  "doc_id": "INV-2025-0045",
  "summary": "Invoice 0045 from Beta Traders for 89.50 EUR, due on 2025-08-28.",
  "key_fields": {
    "vendor_name": "Beta Traders",
    "invoice_number": "0045",
    "invoice_date": "2025-08-14",
    "due_date": "2025-08-28",
    "currency": "EUR",
    "total_amount": 89.50
  },
  "observations": [
    "Early-payment discount mentioned (2%)."
  ],
  "data_quality": {
    "missing_fields": [],
    "suspected_errors": [],
    "confidence_note": "All fields aligned with extracted facts."
  }
}

10) Where to store results

Write JSON summaries to summaries/ and (optionally) a compact CSV view processed/invoice_summaries.csv with doc_id, vendor_name, total_amount, due_date, needs_review, source='bedrock'.
Join this dataset with invoices_wide.csv for dashboard overlays (e.g., a tooltip with the narrative summary).

🗂️ Step 5: Store Final Data Back in S3

1) Folder layout

Use a layout that separates raw vs extracted vs consumable and supports BI tools:

s3://doc-insights-bucket/
  processed/
    wide/                 # final, dashboard-ready “wide” tables (CSV/Parquet)
    kv/                   # merged KV outputs across docs
    tables/               # flattened tables across docs
    expense/              # invoice summaries & line items (if using AnalyzeExpense)
    summaries/            # Bedrock JSON summaries
    parquet/              # same datasets in Parquet (for Athena/QuickSight at scale)
    _staging/             # temp writes (see atomic pattern)
    manifests/            # optional manifest/index files

Partitioning (for Athena/scale):

Prefer hive-style partitions for Parquet:
- processed/parquet/invoices_wide/year=2025/month=09/day=12/part-0001.parquet
Keep CSVs unpartitioned or lightly partitioned by month to avoid too many small files.

2) Naming strategy

Document-scoped files: processed/expense/summary/<doc_id>.csv processed/summaries/<doc_id>.json
Batch/append files: processed/wide/invoices_wide_YYYYMMDD.csv (or append into a single table and de-dupe by doc_id)
Timestamped artifacts: add an ISO timestamp or a monotonically increasing batch id:
- .../2025-09-12T18-40-12Z_invoices_wide.csv
Avoid collisions: include doc_id (or a UUID) in per-doc artifacts; never rely only on original filenames.

3) Atomic write pattern

S3 has strong read-after-write consistency, but clients can still read half-written objects if your writer crashes mid-upload.

Pattern:

Write to processed/_staging/<uuid>/<file.tmp> (or status=writing tag).
Verify checksum/row count.
Copy to final key (e.g., processed/wide/invoices_wide_20250912.csv).
Optionally delete the staging file, or keep with tag status=archived.

Edge cases:

Large files (>5 GB): use multipart upload; always send Content-MD5 for integrity when feasible.
Concurrent writers: add a lightweight lock (S3 object processed/_locks/invoices_wide.lock with short TTL) or write per-writer partitions then compact later.

4) Metadata, tags, and headers

Server-side encryption: SSE-KMS with your CMK (consistent with earlier steps).
Content-Type: set correctly (text/csv, application/json, application/parquet).
Cache-Control: not crucial for data lakes; you can omit or set no-cache for rapidly changing CSVs.
Object tags (helpful for governance/search):
- dataset=invoices, stage=processed, doc_id=<id>, status=ready, source=expense/forms/ocr/bedrock.
Custom metadata (e.g., x-amz-meta-rowcount, x-amz-meta-hash): useful for quick validation.

5) Security & access

Bucket policy: already denies insecure transport; also allow the QuickSight service role to ListBucket and GetObject on processed/*.
KMS key policy: include the QuickSight role principal (not just IAM permissions) so it can decrypt.
Cross-account reads (optional): prefer role assumption + bucket policy + KMS grants.
VPC endpoints: if compute is private, ensure endpoints for S3/KMS are present.

Edge cases:

QuickSight can list but can’t read → key policy is missing the role principal.
Using SPICE: watch dataset size limits; switch to Athena over Parquet when CSVs grow.

6) Data integrity & quality gates

Before moving from _staging → processed:

Schema check: headers exist and match expected column set.
Row count > 0 (or allow zero with a note=empty_batch tag).
Type validation: dates parse (ISO-8601), amounts numeric, currency present.
Sanity checks: if files include subtotal, tax, total, verify subtotal + tax ≈ total (tolerance).
Checksum: compute SHA-256 of the file and store as sidecar:
- processed/wide/invoices_wide_20250912.csv.sha256

7) Lifecycle & retention

Versioning: keep enabled to roll back erroneous overwrites.
Lifecycle rules:
- Transition processed/_staging/ to Expire after 7 days.
- Transition processed/*.csv to STANDARD_IA after 30 days.
- Keep Parquet in STANDARD (or GLACIER if archived) based on access.
S3 Inventory (optional): enable for daily listings & encryption audit.

8) Manifests & discovery (helps BI tools)

Manifest file lists the latest batch outputs:

  {
    "dataset": "invoices_wide",
    "updated_at": "2025-09-12T18:40:12Z",
    "files": [
      "s3://.../processed/wide/invoices_wide_20250912.csv"
    ],
    "row_count": 1243,
    "schema": ["doc_id","vendor_name","invoice_number","invoice_date","due_date","total","currency","needs_review","source","month","year"]
  }

Save as processed/manifests/invoices_wide.latest.json.

Glue Catalog (for Athena): create/update a table pointing to your Parquet path with partitions year, month, day.

9) Eventing

S3 → EventBridge rule on processed/ prefix:
- Trigger a Lambda to refresh QuickSight (or kick an ingestion SPICE refresh).
- Or publish a Slack/Email notification with the manifest summary.
Idempotency: include run_id or batch_id in event detail; ignore duplicates in the consumer.

📊 Step 6: Create a Dashboard in QuickSight

1) Pick your connection path

Path A | S3 CSV (fast start)

Best for small/medium datasets (≤ a few million rows per file).
In QuickSight: Datasets → New dataset → S3.
Point to a single CSV or a manifest JSON (recommended if you have many files).

Path B | Athena + Parquet (scales)

Best for large data, partitions, or frequent refresh.
Store Parquet at processed/parquet/invoices_wide/year=YYYY/month=MM/day=DD/...
In QuickSight: Datasets → New dataset → Athena → select database & table (from Glue Catalog).

Edge cases

If QuickSight is in a different region than your S3 bucket, prefer Athena to avoid cross-region pain.

Using SSE-KMS? Ensure the QuickSight service role is in your KMS key policy and bucket policy.

2) Connect to your data

A) S3 with a manifest (many files)

Create processed/manifests/invoices_wide.latest.json:

{
  "fileLocations": [
    {"URIPrefixes": ["s3://doc-insights-bucket/processed/wide/"]}
  ],
  "globalUploadSettings": {
    "format": "CSV",
    "delimiter": ",",
    "textqualifier": "\"",
    "containsHeader": true
  }
}

Then in QuickSight S3 connector, paste this manifest file’s S3 URL.

B) Athena (Parquet partitions)

Make sure your Glue table points to processed/parquet/invoices_wide/ and has partitions year, month, day.
In QuickSight, choose Direct Query (live) or Import to SPICE (in-memory; faster but size-limited).

Edge cases

“AccessDenied” when previewing = missing KMS permission or bucket grant.

No new data after partition upload = run crawler/add partition OR enable partition projection.

3) Prepare the dataset

In the data prep screen:

Map columns

invoice_number → Text
invoice_date, due_date → Date (format YYYY-MM-DD)
total → Decimal
vendor_name → Text
currency → Text
needs_review → Integer/Boolean
(Optional) ingest_date or processed_at → Date/Time

Create calculated fields (QuickSight expressions)

Month (bucket) truncDate('MM', {invoice_date})
Year extract('YYYY', {invoice_date})
Overdue (flag) ifelse({due_date} < now(), 1, 0)
Processing time (days) – pick the source you actually have:
- If you kept an ingest timestamp: dateDiff('DD', {invoice_date}, {ingest_date})
- Or from S3 key date (if you saved it): dateDiff('DD', {invoice_date}, {s3_ingest_date})
Clean total (fallback) – if some totals are text:

parseDecimal({total})

Edge cases

Mixed date formats? Normalize upstream; otherwise use a calculated field with parseDate() per pattern and coalesce.

Multiple currencies? Keep a currency dimension; optional FX table join later.

4) Build the visuals

A) Total amount by vendor (Top-N bar)

Visual: Horizontal bar
X: sum({total})
Y: vendor_name
Sort: by sum(total) desc
Filter: Top 10 vendors (add a Top N filter)
Optional: color by needs_review to spot messy vendors

B) Monthly invoice totals (trend)

Visual: Line chart
X: truncDate('MM', {invoice_date})
Y: sum({total})
Forecast (optional): enable forecast to project next months
Tooltip: add countDistinct({invoice_number}) for volume context

C) Average processing time (days)

Visual: KPI or bar by vendor
Field: avg(dateDiff('DD', {invoice_date}, {ingest_date}))
Group by: vendor_name (or monthly bucket)
Filter: exclude needs_review = 1 if you want only clean rows

D) Bonus quick wins

Overdue heatmap: vendor_name × month with color sum(ifelse({due_date} < now(), {total}, 0))
Quality gate: table filtered to needs_review = 1 (your manual review queue)

5) Filters, controls, and parameters

Date range control: Filter on {invoice_date} with “Relative dates” (e.g., Last 12 months).
Vendor control: Dropdown on vendor_name (multi-select).
Confidence/quality toggle: needs_review = 0 by default (checkbox control).
Currency control (if applicable): filter or parameter to select a single currency.

Edge cases

If your dataset mixes currencies, don’t sum across different codes in one visual unless you convert first.

6) SPICE vs Direct Query

SPICE (Import): fastest visuals, schedule refreshes (e.g., hourly). Watch capacity limits.
Direct Query: always live; best with Athena/Parquet; slightly slower per query.

Recommended

Start with SPICE for S3 CSVs; switch to Athena + SPICE as you scale.

7) Refresh strategy

S3 CSV path
- If you overwrite one key (e.g., invoices_wide_latest.csv), set a refresh schedule (hourly/daily).
- If you append new files, use a manifest and schedule refresh.
Athena path
- After adding partitions, either run a crawler or use partition projection.
- In QuickSight, schedule SPICE refresh or stay on Direct Query.

Edge cases

Refresh fails with KMS errors → add QuickSight role to CMK key policy.

Stale data → you’re writing to new paths; either point QS at a stable key or use manifests.

8) Sharing, RLS, and governance

Share analysis → Publish dashboard; grant users/groups access.
Row-Level Security (RLS): upload an RLS mapping dataset (user, vendor_name) to restrict rows per viewer.
Data dictionary tab: add field descriptions so the team knows what “processing time” means.

10) Minimal recipe

Connect

S3: dataset → S3 → manifest → validate preview.
Athena: dataset → Athena → select table.

Model

Types: map date/number fields.
Calcs: month = truncDate('MM', {invoice_date}), processing_days = dateDiff('DD', {invoice_date}, {ingest_date}).

Visualize

Bar: sum(total) by vendor_name (Top 10).
Line: sum(total) by month.
KPI/Bar: avg(processing_days) by vendor.

Control

Date range (last 12 months).
Vendor multi-select.
needs_review toggle (default 0).

Operationalize

SPICE refresh schedule (hourly/daily).
If Athena: ensure partitions & permissions.

💭 Conclusion

This was honestly a fun little fortnight project.
It saved me hours of manual work and gave me clean, visual summaries of everything I needed. By combining S3, Textract, Bedrock, and QuickSight, you can turn unstructured, scanned documents into analysis-ready data and clear dashboards. This pipeline handles extraction, normalisation, summarisation, governed storage, and visualisation, built with security (SSE-KMS), least-privilege IAM, and scalable patterns (async Textract, Parquet/Athena). It replaces manual review with consistent, auditable automation and surfaces vendor spend, monthly trends, and processing KPIs in minutes. From here, you can harden orchestration with Step Functions, add Row-Level Security in QuickSight, and extend the same approach to receipts, contracts, or reports. Small lift-big leverage.

You could easily apply this to:

Legal contracts
Academic papers
Medical prescriptions
Tax docs

If you try it out and get stuck, drop a comment, and I'd be happy to help.

HappyHacking 👋

🚀 Scaling React Like Big Tech: Folder Structures, Clean Code & Beyond

Mursal Furqan Kumbhar — Sat, 30 Aug 2025 18:55:22 +0000

Hey Everyone 👋

I know it's been quite a long gap since my last article. But we are finally back with a bang.

So if you’ve ever wondered how Airbnb, Shopify, or Meta keep their codebases sane while thousands of engineers ship features daily — the answer is simple: structure + discipline + automation.

This isn’t about writing code that just works. It’s about writing code that:

Scales with your team.
Survives onboarding of 10 new devs.
Passes reviews without a hundred nitpicks.
And doesn’t make you cry when you open it six months later.

Let’s break down what the big companies do differently — and how you can bring those practices into your own projects.

📂 Folder Structure that Scales

Top companies don’t just throw files into a random components/ folder and hope for the best. They use feature-driven architecture, which grows naturally with the product.

❌ Traditional (Bad) Structure

src/
├── components/
│   ├── Button.jsx
│   ├── Card.jsx
│   ├── Form.jsx
├── services/
├── hooks/
└── App.jsx

This looks fine at first… until your app has 300+ components and no one knows which belongs to what.

✅ Scalable (Feature-Based) Structure

src/
├── features/
│   └── auth/
│       ├── components/
│       ├── hooks/
│       └── services/
├── shared/
│   ├── ui/
│   └── utils/
└── App.jsx

📌 Key Difference: Everything related to a single feature (Auth, Products, Cart, etc.) is grouped. Shared logic/components live in shared/.

Approach	Pros	Cons
Traditional (by type)	Simple to start	Becomes chaos at scale
Feature-Based	Easy ownership, scalable, testable	Slightly more setup upfront

🎨 Design Systems & UI Libraries

Big companies don’t reinvent the button every Tuesday. They use design systems and UI libraries.

🚀 Airbnb → Lona
🚀 Shopify → Polaris
🚀 Meta → Reusable UI + strict guidelines

👉 Tools you can use: Chakra UI, Tailwind + Headless UI, Radix UI

Visual Example

❌ Without Design System:

<Button variant="danger" color="red" bg="red" />

✅ With Design System:

<DeleteButton />

Why Use Design Systems?	Benefits
Consistency	UI looks the same across app
Speed	Devs move faster
Accessibility	Built-in ally support
Brand Identity	Product feels cohesive

🧑‍💻 Hooks = Where Logic Lives

Business logic doesn’t belong inside components. Custom hooks make code reusable, testable, and clean.

Example:

// useProduct.js
export function useProducts() {
   const [data, setData] = useState([])
   useEffect(() => {
      fetchProducts().then(setData)
   }, [])
   return data
}

✅ UI stays declarative.
✅ Logic stays modular.

📊 Diagram: Components vs Hooks

[Component] ---> UI (render)
       |
       v
 [Custom Hook] ---> Data Fetching, State, Business Logic

🧼 Clean Code = Scalable Code

Big tech engineers follow clean code principles:

SRP → Single Responsibility Principle
KISS → Keep It Simple, Stupid
DRY → Don’t Repeat Yourself

❌ Bad	✅ Good
`<Button variant="danger" color="red" bg="red" />`	`<DeleteButton />`

✅ Abstraction = Fewer Bugs + Cleaner Reviews

🔍 Real-Time Code Reviews + Linting

At big companies, automation + reviews are mandatory.

✅ Code Reviews → shared knowledge
✅ Pre-Commit Hooks (Husky) → catch errors early
✅ ESLint + Prettier → consistent style

CI/CD Pipeline Visual

[Developer Pushes Code] 
       |
       v
[Pre-Commit Checks: Lint + Tests]
       |
       v
[Pull Request Review]
       |
       v
[CI/CD: Build + Deploy + Monitor]

🔐 Security & ♿ Accessibility First

Security isn’t optional. Accessibility isn’t either.

Security Checklist

Prevent XSS and CSRF
Proper token handling
Input validation

Accessibility Checklist

Keyboard navigation works
Screen reader support
High color contrast

👉 Use tools like axe-core, eslint-plugin-jsx-ally, lighthouse.

⚡ Automation + Monitoring

Big teams don’t just ship code — they ship pipelines + monitoring.

✅ CI/CD Pipelines (GitHub Actions, Vercel, CircleCI)
✅ Bundle Size Alerts
✅ Error Tracking (Sentry, Datadog)
✅ Performance Monitoring (Lighthouse CI)

📊 Example:

Commit → Build → Test → Deploy → Monitor

📊 Documentation & Knowledge Sharing

What separates side projects from real companies?
👉 Documentation.

Good README.md = contributor-friendly.
Internal wikis = no lost knowledge.
ADRs (Architecture Decision Records) = explain why choices were made.

🧠 Culture = The Invisible Architecture

Finally — even the best code structure won’t save a bad culture. Big companies win because:

PRs = conversations, not fights.
Automation is embraced, not skipped.
Docs + testing = part of “done”.

At the end of the day, how your team codes together matters more than folder names.

🚦 Startup vs Big Tech Codebases

Here’s how things usually look in a scrappy startup vs how big tech organizes for scale:

┌────────────────────┐           ┌─────────────────────────┐
│   Startup Style    │           │     Big Tech Style      │
├────────────────────┤           ├─────────────────────────┤
│ 📂 components/     │           │ 📂 features/auth/       │
│ 📂 hooks/          │           │   ├── components/       │
│ 📂 services/       │           │   ├── hooks/            │
│ App.jsx            │           │   └── services/         │
│                    │           │ 📂 shared/ui/           │
│ Easy to start ✅   │           │ 📂 shared/utils/        │
│ Chaos later ❌      │           │ Scales across teams ✅  │
└────────────────────┘           └─────────────────────────┘

📊 Side-by-Side Table

Aspect	Startup-Style	Big Tech-Style
Folder Structure	Flat, all components in one folder	Feature-based, modular
UI	Custom each time	Design system + UI library
Logic	Inside components	Extracted into hooks
Code Quality	Quick fixes, less abstraction	Clean code, reusable patterns
Reviews	Ad-hoc, sometimes skipped	Mandatory PRs + linting + tests
Security	Later concern	Built-in from day one
Docs	README only	Wikis + ADRs + onboarding guides
Culture	“Move fast, break things”	“Move fast, don’t break prod”

🎯 Final Takeaway

Big companies scale not because their engineers are magical unicorns 🦄, but because they follow battle-tested practices:

Organize code by features, not file types.
Use design systems for UI.
Extract logic into hooks.
Follow clean code principles.
Automate reviews + pipelines.
Prioritize security & accessibility.
Write documentation.
Build a team culture around consistency.
Startups often optimize for speed, but end up with technical debt.
Big Tech optimizes for scalability, making onboarding, reviews, and growth smooth.
The sweet spot? Start with startup speed, but gradually adopt Big Tech practices before chaos hits.

Happy Hacking 👋

WSL2 TensorFlow GPU Setup – RTX 4060 + Ubuntu 22.04 + CUDA 12.2 + cuDNN

Mursal Furqan Kumbhar — Wed, 16 Apr 2025 16:36:12 +0000

🛠 Prereqs:

WSL2 enabled on Windows
NVIDIA GPU driver (≥ v535) installed on Windows
Ubuntu 22.04 installed via WSL
Download cuDNN manually while logged in to NVIDIA Developer

🧱 1. Setup Python environment in Ubuntu (WSL2)

sudo apt update && sudo apt install python3.11 python3.11-venv gcc-11 g++-11 -y
python3.11 -m venv ~/tf-gpu-311
source ~/tf-gpu-311/bin/activate
pip install --upgrade pip
pip install tensorflow==2.15.0

⚙️ 2. Install CUDA 12.2 (runfile method)

wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda_12.2.0_535.54.03_linux.run
chmod +x cuda_12.2.0_535.54.03_linux.run
sudo ./cuda_12.2.0_535.54.03_linux.run

❌ Uncheck driver install (you're using Windows one)
✅ Install toolkit only

Then set up environment:

echo 'export PATH=/usr/local/cuda-12.2/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

📦 3. Install cuDNN 8.9.4 (for CUDA 12.x)

Go to cuDNN Downloads
Login → Choose:
- Version: cuDNN 8.9.4
- OS: Linux
- Installer: Local Installer for Ubuntu22.04 x86_64 (.deb)
Download .deb file and move it into Ubuntu:

cp /mnt/c/Users/<yourname>/Downloads/cudnn-local-repo-ubuntu2204-8.9.4.25_1.0-1_amd64.deb ~/
cd ~
sudo dpkg -i cudnn-local-repo-ubuntu2204-8.9.4.25_1.0-1_amd64.deb
sudo cp /var/cudnn-local-repo-ubuntu2204-8.9.4.25/cudnn-*-keyring.gpg /usr/share/keyrings/
sudo apt update
sudo apt install libcudnn8 libcudnn8-dev -y

🔄 4. Restart WSL2

In PowerShell (Windows):

wsl --shutdown

Then open Ubuntu again:

source ~/tf-gpu-311/bin/activate

✅ 5. Test TensorFlow GPU

import tensorflow as tf
print("TensorFlow version:", tf.__version__)
print("GPUs available:", tf.config.list_physical_devices('GPU'))
print("CUDA enabled:", tf.test.is_built_with_cuda())
print("GPU name:", tf.test.gpu_device_name())

💚 Expected Output

GPUs available: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
CUDA enabled: True
GPU name: /device:GPU:0

Happy TensorFlowing, GPU wizard 🧙‍♂️⚡

Smart Image Tagging on AWS - The Finale

Mursal Furqan Kumbhar — Fri, 04 Apr 2025 12:10:00 +0000

Hello and Greetings Everyone 👋

I am hoping that you are having fun in our journey of creating the auto-tagging application for images like a pro using Amazon Rekognition, AWS Lambda and S3.

Recap

Let's do a quick recap of what we did in the last two parts.

1. Auto-Tag Images on AWS using Amazon Rekognition + Lambda + S3

Created an S3 bucket to store uploaded images
Created a Lambda function to handle image upload events
Set up IAM roles and permissions for Rekognition and S3 access
Used Amazon Rekognition to detect objects/scenes in images
Logged detected labels to CloudWatch Logs
Connected S3 and Lambda using event notifications

2. Level Up Your Auto-Tagging Pipeline on AWS

Uploaded test images and successfully triggered Lambda
Viewed detection results in CloudWatch logs
Improved the Lambda logic to store Rekognition labels as .json files inside a tags/ folder in S3
Validated that the full pipeline was working end-to-end

I guess that would have jogged your memory. In this article, we are going to discuss:

Sending real-time email notifications using Amazon SNS
Wrapping up the entire series with final thoughts and next steps

Let's Go 🚀

Send Real-Time notifications via Amazon SNS

As discussed, we already have created the pipeline for auto-tagging the image and saving the data in a JSON file. Now what can be cooler than getting an email whenever it happens? In this step, we will modify and enhance our pipeline by integrating Amazon SNS to send notifications everytime an image is tagged.

Create an SNS Topic

Like every other AWS Service, we need to first go to the AWS Console and find SNS (Simple Notification Service). On the SNS Homepage, write your topic's name, ImageTaggingNotifications, in the Create Topic textbox (As shown in the figure below). Once done, click on Next Step button. In the next screen, select Standard as the type and click Create Topic at the bottom of the screen leaving everything default.

Seeing our friend, the green notification box would tell us that our topic is created. Once done, click on orange Create Substription button and fill in Email under Protocol and type in your email under Endpoint.

All done? Click on Create Subscription. As soon as subscription is created, check your email for the confirmation from AWS. Click on the link to confirm the subscription.

Remember: Don't skip/miss this step or you won't get any emails.

The email would look something like this:

Give Lambda Permission to Publish to SNS

After successfully creating the topic and subscribing to it with your email, we need to go to IAM > Roles > RekognitionLambdaRole > Add Permissions > Create Inline Policy.

In the next screen, click on JSON and copy paste the following code:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:your-region:your-account-id:ImageTaggingNotifications"
    }
  ]
}

Don't forget to replace your-region (e.g. us-east-1) and your-account-id (e.g. 123456789012) accordingly

Click next and Give it a good name such as AllowSnsPublishPolicy in the next screen. After doing so, click on Create Policy to finish the policy creation process.

Update the Lambda Code to Send Notifications

After creating the policy, we need to let the Lambda function know when and how to send the notification. To do so, we need to the below given code right after saving the .json file in S3:

import boto3

sns = boto3.client('sns')
TOPIC_ARN = 'arn:aws:sns:us-east-1:896415180455:ImageTaggingNotifications'  # Replace with your actual ARN

# Get top 3 labels
top_labels = label_data[:3]
labels_text = ', '.join([f"{l['Name']} ({l['Confidence']}%)" for l in top_labels])
message = f"Image '{key}' was tagged with:\n\n{labels_text}"

# Send SNS notification
sns.publish(
    TopicArn=TOPIC_ARN,
    Message=message,
    Subject='[Image Tagging Alert] New Image Tagged on S3'
)

Or if you didn't get it, replace the complete code for Lambda Function with:

import json
import boto3
import urllib.parse
import datetime

rekognition = boto3.client('rekognition')
s3 = boto3.client('s3')
sns = boto3.client('sns')

# Replace with your actual SNS topic ARN
TOPIC_ARN = '<TOPIC-ARN>'

def lambda_handler(event, context):
    print("Lambda Triggered at:", str(datetime.datetime.now()))
    print("Received event:", json.dumps(event, indent=2))

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    # ✅ Skip non-image files
    if not key.lower().endswith(('.jpg', '.jpeg', '.png')):
        print(f"Skipping file {key} — not a supported image format.")
        return {
            'statusCode': 200,
            'body': json.dumps('Skipped non-image file.')
        }

    try:
        # 🔍 Step 1: Detect labels using Rekognition
        response = rekognition.detect_labels(
            Image={
                'S3Object': {
                    'Bucket': bucket,
                    'Name': key
                }
            },
            MaxLabels=10,
            MinConfidence=80
        )

        # 📋 Step 2: Format label data
        label_data = []
        for label in response['Labels']:
            label_data.append({
                'Name': label['Name'],
                'Confidence': round(label['Confidence'], 2)
            })
            print(f" - {label['Name']} ({label['Confidence']:.2f}%)")

        # 💾 Step 3: Save labels as JSON to /tags/
        image_name = key.rsplit('/', 1)[-1].rsplit('.', 1)[0]
        tags_key = f"tags/{image_name}.json"

        s3.put_object(
            Bucket=bucket,
            Key=tags_key,
            Body=json.dumps(label_data, indent=2),
            ContentType='application/json'
        )

        print(f"Saved tags JSON to: {tags_key}")

        # 📬 Step 4: Send SNS notification with top labels
        top_labels = label_data[:3]
        labels_text = ', '.join([f"{l['Name']} ({l['Confidence']}%)" for l in top_labels])
        message = f"Image '{key}' was tagged with:\n\n{labels_text}"

        sns.publish(
            TopicArn=TOPIC_ARN,
            Message=message,
            Subject='[Image Tagging Alert] New Image Tagged on S3'
        )

        print("Notification sent via SNS!")

        return {
            'statusCode': 200,
            'body': json.dumps('Labels detected, saved to S3, and notification sent!')
        }

    except Exception as e:
        print(f"Error processing {key}: {str(e)}")
        raise e

Replace TOPIC_ARN with your actual SNS Topic ARN.

And it's done!

After this, our entire email notification pipeline is complete. Let's test this. To test, go to your AWS S3 bucket and upload the image like we did in the previous article. If the entire pipeline is working, you will immediately receive an email with the top 3 tags of the image in the email address you gave while creating the SNS Topic. The email would look something like this:

If you don't get the email, go through all the process and see what you missed. And in case you don't find what you did wrong, write me in the comments below.

🎉 Conclusion

And that’s a wrap, folks! 🎬
We’ve come a long way — from uploading images to S3 and detecting labels with Rekognition, to storing results, sending real-time notifications, and polishing our code like true builders 💪

This serverless, scalable, and smart image tagging workflow is just one example of how you can combine AWS Lambda, Amazon S3, Rekognition, and SNS to automate powerful tasks with ease.

Whether you’re building this for fun, a portfolio project, or the foundation of a real-world app — I hope this series inspired you to explore what’s possible on AWS.

Thanks for sticking around till the end — and as always, if you enjoyed this, drop a ❤️, share it, or leave a comment.

Until next time... Ciao! 👋

Happy Coding 🧑‍💻

Level Up Your Auto-Tagging Pipeline on AWS

Mursal Furqan Kumbhar — Thu, 03 Apr 2025 12:05:00 +0000

Hello and Greetings Everyone 👋

I hope you would have loved reading and experimenting with creating your Auto-Tag application using Amazon Rekognition, AWS Lambda and S3 here.

It's a new day, and we don't want to leave everything hanging. Right? 😀 As promised, in this article we are going to discuss:

Uploading a Test Image to S3
View Results in CloudWatch Logs
Save Tags in a JSON file in S3

Without wasting much of our time, Let's go 🚀!

Uploading a Test Image to S3

Since we have already created our basic pipeline with S3 Bucket, Lambda Function and Rekognition Pipeline, it's time to put everything to the test. We are now going to upload a sample image, and pray that our system detects and logs the labels, automatically 😁

Choose a Test Image

If you remember clearly, we added the suffixes in our system, so we now have to be careful about the extension of our images. We can pick any .jpg, .jpeg or .png image. You can either pick a photo from your phone/computer, get a sample image from the internet or any other source, but remember to keep it small and simple for quick processing since we are still on the testing phase.

Upload via AWS Console

Now that we have selected the image, we are going to upload it on our S3 bucket, using AWS Console. I hope you haven't logged out of the AWS Portal (If yes, sign in again...)

Go to your S3 Bucket (for me, image-tagger-mursal-april-five)
Click the Upload button and click Add files.

Browse to your image in your mobile phone/computer and choose it. You will see your uploaded image in the directory.
Leave all the other settings as default and directly click Upload.

Once it's done, click Close

Doing this will immediately trigger ou lambda function behind the scenes. Once the image is uploaded, Lambda will Pick up the image from S3 and send it to Amazon Rekognition to detect objects and scenes. It will also log everything to CloudWatch in the end. Let's see what's bee logged for our image (Excited? 😜)

View Detected Labels in CloudWatch Logs

Now that our image is uploaded, we need to see how and what Rekognition detected. Well, that's where Amazon CloudWatch Logs jumps in. Everytime Lambda runs, it will log the entire output, including the labels, to CloudWatch. Let's see that.

Navigate to CloudWatch Logs

In our AWS Console, search for CloudWatch in the services menu and open it and click Logs in the left sidebar, followed by Log groups. We will be looking for a log group named something like:

/aws/lambda/AutoTagImageFunction

Once inside, click that log group and click the most recent log stream (Check for your upload timestamp). Once inside the log stream, we will find logs like:

Received event: {
  "Records": [
    {
      ...
      "s3": {
        "bucket": {
          "name": "image-tagger-mursal-april-five"
        },
        "object": {
          "key": "sample.jpg"
        }
      }
    }
  ]
}

Followed by the recognition output:


2025-04-03T05:30:46.444Z
Labels detected for WhatsApp Image 2025-03-23 at 16.28.43_45962656.jpg:
2025-04-03T05:30:46.444Z
- Adult (99.23%)
2025-04-03T05:30:46.444Z
- Male (99.23%)
2025-04-03T05:30:46.444Z
- Man (99.23%)
2025-04-03T05:30:46.444Z
- Person (99.23%)
2025-04-03T05:30:46.444Z
- Standing (98.34%)
2025-04-03T05:30:46.444Z
- Head (98.21%)
2025-04-03T05:30:46.444Z
- Face (97.73%)
2025-04-03T05:30:46.444Z
- Smile (97.26%)
2025-04-03T05:30:46.444Z
- Jacket (96.45%)
2025-04-03T05:30:46.444Z
- Portrait (95.66%)

Save Detected Labels as a JSON File in S3

At the current moment, we are logging all the labels to CloudWatch, but we are going to store those labels in a tags/ folder in our existing S3 bucket since we need to record each image. This is going to be our intended folder structure:

image-tagger-mursal-april-five/
├── WhatsApp Image 2025-03-23 at 16.28.43_45962656.jpg
└── tags/
    └── WhatsApp Image 2025-03-23 at 16.28.43_45962656.json

Remember: Create your tags folder before moving forward.

For that, we need to update our Lambda Code:

import json
import boto3
import urllib.parse

rekognition = boto3.client('rekognition')
s3 = boto3.client('s3')

def lambda_handler(event, context):
    print("Received event:", json.dumps(event, indent=2))

    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    try:
        # Step 1: Detect labels
        response = rekognition.detect_labels(
            Image={
                'S3Object': {
                    'Bucket': bucket,
                    'Name': key
                }
            },
            MaxLabels=10,
            MinConfidence=80
        )

        print(f"Labels detected for {key}:")
        label_data = []
        for label in response['Labels']:
            label_data.append({
                'Name': label['Name'],
                'Confidence': round(label['Confidence'], 2)
            })
            print(f" - {label['Name']} ({label['Confidence']:.2f}%)")

        # Step 2: Save labels as JSON to /tags/
        tags_key = f"tags/{key.rsplit('/', 1)[-1].rsplit('.', 1)[0]}.json"
        s3.put_object(
            Bucket=bucket,
            Key=tags_key,
            Body=json.dumps(label_data, indent=2),
            ContentType='application/json'
        )
        print(f"Saved tags JSON to {tags_key}")

        return {
            'statusCode': 200,
            'body': json.dumps('Labels detected and saved to S3!')
        }

    except Exception as e:
        print(f"Error processing {key}: {str(e)}")
        raise e

Once done, test and Deploy the new code in our Lambda Function. Let's test it.

Upload a new Image to S3
Make sure the image is either .jpg, .jpeg or .png format.
This will trigger the Lambda Function
Lambda Function will have Amazon Rekognition detect the labels
All the data will be saved in a JSON file inside the tags folder that you created earlier

Now your tags folder looks like this:

Click on your relevant .json file to see the results. Here you will find all the data of your uploaded file.

Click on Open or Download to access the file. Your JSON file have an output similar to the one below:

[
  {
    "Name": "Bag",
    "Confidence": 95.62
  },
  {
    "Name": "Document",
    "Confidence": 85.53
  },
  {
    "Name": "Receipt",
    "Confidence": 85.53
  },
  {
    "Name": "Text",
    "Confidence": 85.53
  },
  {
    "Name": "Plastic",
    "Confidence": 81.17
  },
  {
    "Name": "Plastic Bag",
    "Confidence": 81.17
  }
]

BONUS: You can also try to see your JSON file from the terminal with the following command:

aws s3 cp s3://<bucket-name>/tags/<file-name>.json .
cat <file-name>.json

Remember: Don't forget to replace s3:///tags/ and with the relevant names.

You did it!

Still hanging in there? You’re a legend! 😎
That wraps up Part 2 of our Smart Serverless Image Tagging on AWS series. This time, we went beyond the basics — tested the full automation flow, explored CloudWatch logs, and even saved our Rekognition results as structured .json files directly in S3. Clean. Serverless. Smart. 💡

In the next part, we’ll level it up with some bonus magic:

Email notifications via SNS 📬
Code modularization 🧩
Maybe even a simple front-end to display your tagged images 👀

Until then, keep building cool stuff — and as always, hit me up in the comments if you have any ambiguity! 😜

Ciao 👋

Auto-Tag Images on AWS using Amazon Rekognition + Lambda + S3

Mursal Furqan Kumbhar — Wed, 02 Apr 2025 18:24:15 +0000

Hello and Greetings 👋 Everyone!

I know it's been quite a gap since my last article and I sincerely apologize for the gap. But I am back with a bang 😉

We all have been to a situation where we want create something to make our lives easier in daily routine. RIGHT? If you have ever wanted to build a smart system, that automatically tags images the moment they're uploaded (Like Google Photos or Amazon Photos)? Wanted to create an Application to upload vacay pictures and instantly tells you what you are looking at? Trees, Mountains, People, Emotions? That too without writing any Machine Learning Model yourself?

In this article, we are going to build a fully serverless pipeline that'll:

Let the users upload an image to an S3 Bucket
It will trigger a Lambda Function automatically
Will use Amazon Rekognition to detect objects/scenes
Store the generated labels or tags into DynamoDB or, if you want, write them to an S3 JSON file
Send a notification via SNS or Email

I know it's quite a mouthful and therefore we will be doing a series of articles on this process. We'll need:

Tool	Usage
Amazon S3	Store Images
Amazon Rekognition	Object and Scene Detection
AWS Lambda	Adding everything together
DynamoDB or S3 JSON	Store the Results

Let's Go...! 🚀

Step 1: Create an S3 Bucket and Enable Event Notifications

In this step, all you need to do is take your laptop or start your workstation, login to your OS, open your browser and go to our favorite portal, i.e. AWS Portal. Sign-in and begin by creating an S3 Bucket. We will use this bucket so that the users can upload their images. This bucket will trigger a lambda function everytime a new image upload is detected.

Create S3 Bucket

Simply open your AWS Console/CLI, and wait until the environment is ready to listen to you.

Then simply write this

aws s3api create-bucket --bucket unique-bucket-name --region eu-west-3

Remember to use a unique bucket name at 'unique-bucket-name' & use your preferred region.

Create RekognitionLambdaRole

Having created our S3 bucket, we need to create the IAM Role before moving forward. Simply follow these steps to do so:

Go to IAM > Roles > Create Role

Trusted Entity:
- Choose AWS Service
- Use Case: Lambda
- Click Next
In the next screen, attach the following policies and click Next:
- AmazonRekognitionFullAccess
- AmazonS3ReadOnlyAccess
- CloudWatchLogsFullAccess
In the next screen, name your role:
- Role Name: RekognitionLambdaRole
- Click Create Role

Our new Role is ready!

Create Lambda Function

Before moving forward, we need to create our Lambda Function, attach the correct IAM role, write it's python code and deploy it.

Go to AWS Lambda Console, and click create function.

Fill in the following details:

Author From Scratch
Function Name: AutoTagImageFunction
Runtime: Python 3.12 or if a newer version available
Permissions:
- Select "Use an Existing Role"
- Choose the role we created earlier, i.e. RekognitionLambdaRole

Click Create Function to complete the function creation process.

Congratulations 🎉 Our Lambda Function is done. But wait. We are not done with the Lambda Function yet.

In the next screen, replace the default code with the following python code in the code editor:

import json
import boto3
import urllib.parse

rekognition = boto3.client('rekognition')

def lambda_handler(event, context):
    print("Received event:", json.dumps(event, indent=2))

    # Get the bucket and object key
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'])

    try:
        # Call Rekognition
        response = rekognition.detect_labels(
            Image={
                'S3Object': {
                    'Bucket': bucket,
                    'Name': key
                }
            },
            MaxLabels=10,
            MinConfidence=80
        )

        print(f"Labels detected for {key}:")
        for label in response['Labels']:
            print(f" - {label['Name']} ({label['Confidence']:.2f}%)")

        return {
            'statusCode': 200,
            'body': json.dumps('Labels detected and logged!')
        }

    except Exception as e:
        print(f"Error processing {key}: {str(e)}")
        raise e

The next step is to Deploy and Test.
Simply click Deploy to save the function. We will test this function later.

Enable Event Notifications for Uploads

Now we have our S3 bucket created. Let's go for the next step, i.e. triggering the Lambda Function on S3 Uploads whenever a new .jpeg image is uploaded to your bucket. Remember: We want it to automatically trigger a lambda function. Here, we need two things:

Lambda Function ARN (Will get this after this step is done)
Permission for S3 to trigger our Lambda Function

First go to the s3 bucket that we just created.

And navigate to the properties tab.

Scroll to 'Event Notifications' and Click 'Create Event Notification'

Fill in the following Details:

Name: onImageUpload
Suffix Filter: Add .jpg, .jpeg, .png (You can leave it all to allow all images)
Event Type: PUT
Destination: Select Lambda Function
- Choose your lambda function that we created earlier.

Having done all that, save the changes by clicking Save Changes. Once that is done, we need to add permissions as well [REEEAALLLYY IMPORTANT ⚠️]

Aft the lambda function is created, run the following command in the CLI to allow S3 to invoke it.

aws lambda add-permission \
  --function-name AutoTagImageFunction \
  --principal s3.amazonaws.com \
  --statement-id S3InvokePermission \
  --action "lambda:InvokeFunction" \
  --source-arn arn:aws:s3:::image-tagger-mursal-april-five

Make sure you replace AutoTagImageFunction with your actual Lambda function name (if it's different 😉)
Don't forget to replace image-tagger-mursal-april-five with your actual S3 bucket name.

Tired Already?

Well, that's it for Part 1 of our Smart Serverless Image Tagging on AWS series. In this article, we have set-up the basic pipeline, i.e. S3, Lambda, Rekognition, and enabled them to auto-detect labels from the uploaded images. In the next article, we are going to cover:

Uploading test images to S3
Viewing Label results in CloudWatch
Saving tags to a JSON file in S3
Sending Notifications via SNS

I guess we are going great. Let me know in the comments section if you guys want more of the AWS Stuff 😜

Ciao 👋

Introduction | Graph Neural Networks (GNNs) & Knowledge Graphs on AWS

Mursal Furqan Kumbhar — Sat, 15 Mar 2025 08:18:16 +0000

The rapid advancements in artificial intelligence (AI) have unlocked new ways to process and learn from complex, interconnected data. While traditional deep learning models excel at structured and unstructured data, they struggle to capture relationships between entities. This is where Graph Neural Networks (GNNs) and Knowledge Graphs come in—offering a powerful way to model dependencies, enhance reasoning, and improve AI predictions.

In this article series, we’ll explore how AWS provides scalable infrastructure for building, training, and deploying GNN-based AI models. From Amazon Neptune for knowledge graph storage to SageMaker for graph-based ML, we’ll dive into practical implementations and real-world use cases such as fraud detection, recommendation systems, and AI-powered search engines.

This first article will introduce the core concepts of GNNs, why they matter, and how AWS enables scalable Graph AI. Let’s get started! 🚀

What are Graph Neural Networks (GNNs)?

Graph Neural Networks (GNNs) are a class of deep learning models designed to process graph-structured data. Unlike traditional neural networks that work on structured tabular data or unstructured images and text, GNNs can learn representations from nodes, edges, and their relationships in a graph.

A graph consists of:

Nodes (Vertices): Entities in the dataset (e.g., users, products, molecules).
Edges: Relationships between entities (e.g., social connections, molecular bonds).
Features: Attributes associated with nodes and edges (e.g., user preferences, product categories).

GNNs use message passing mechanisms to propagate information across the graph, enabling learning at both the node-level (e.g., predicting node properties), edge-level (e.g., link prediction), and graph-level (e.g., graph classification).

Importance of Graph-Based Learning in AI

Graph-based learning has seen widespread adoption in various AI-driven domains:

1. Social Network Analysis

Use Case: Predicting social connections (e.g., Facebook friend recommendations).
GNN Role: Learning user embeddings based on mutual friends and shared interactions.

2. Fraud Detection in Financial Transactions

Use Case: Detecting fraudulent transactions in banking and e-commerce.
GNN Role: Identifying unusual patterns in transaction networks.

3. Recommender Systems (Graph-Based Search & Recommendation)

Use Case: Enhancing movie, product, and content recommendations (e.g., Netflix, Amazon).
GNN Role: Capturing relationships between users and items to improve recommendations.

4. Drug Discovery & Bioinformatics

Use Case: Predicting molecular interactions for drug design.
GNN Role: Learning molecular structures and identifying potential drug candidates.

5. Knowledge Graphs for NLP & AI Search

Use Case: Enhancing Large Language Models (LLMs) with knowledge graphs.
GNN Role: Structuring factual knowledge for better AI reasoning and retrieval.

How AWS Provides Scalable Infrastructure for Graph ML

AWS offers a robust suite of services that support GNN-based AI applications, providing storage, training, and deployment capabilities.

1. Amazon Neptune (Managed Graph Database for Knowledge Graphs)

A fully managed graph database optimized for high-performance querying.
Supports Gremlin, SPARQL, and openCypher for flexible graph analytics.
Enables real-time graph-based search for fraud detection, recommendations, and knowledge graphs.

2. Amazon SageMaker (For Training & Deploying GNN Models)

Provides pre-built environments for deep learning frameworks like PyTorch Geometric (PyG) and Deep Graph Library (DGL).
Supports distributed training for large-scale graph processing.
Enables MLOps pipelines for graph-based AI models.

3. AWS Glue (Data Preprocessing for Graph ML)

Used to clean, transform, and link data before ingesting into Neptune.
Supports automated ETL (Extract, Transform, Load) for complex graph datasets.

4. Amazon OpenSearch (Vector Search + Graph Embeddings for AI)

Stores graph-based embeddings generated by GNN models.
Supports hybrid retrieval (vector search + graph relationships) for AI-powered search.

5. AWS Lambda & Step Functions (Serverless Graph AI Pipelines)

Automates graph-based workflows for real-time fraud detection and recommendation systems.
Enables event-driven graph learning when combined with Kinesis and Neptune Streams.

Graph Neural Networks Implementation on AWS (Code Example)

Let’s train a basic GNN using Amazon SageMaker with Deep Graph Library (DGL) on AWS.

Step 1: Install Dependencies

!pip install torch dgl boto3 sagemaker

Step 2: Load a Sample Graph Dataset

import dgl
import torch
import torch.nn as nn
import torch.optim as optim
import dgl.data

# Load a sample graph (Cora dataset)
dataset = dgl.data.CoraGraphDataset()
graph = dataset[0]

# Check graph structure
print(f"Number of nodes: {graph.num_nodes()}")
print(f"Number of edges: {graph.num_edges()}")

Step 3: Define a GNN Model

class GCN(nn.Module):
    def __init__(self, in_feats, hidden_size, num_classes):
        super(GCN, self).__init__()
        self.conv1 = dgl.nn.GraphConv(in_feats, hidden_size)
        self.conv2 = dgl.nn.GraphConv(hidden_size, num_classes)

    def forward(self, g, features):
        x = self.conv1(g, features)
        x = torch.relu(x)
        x = self.conv2(g, x)
        return x

Step 4: Train the Model

# Get feature and label data
features = graph.ndata['feat']
labels = graph.ndata['label']

# Initialize model
model = GCN(features.shape[1], 16, dataset.num_classes)
optimizer = optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()

# Training loop
for epoch in range(50):
    model.train()
    logits = model(graph, features)
    loss = loss_fn(logits, labels)

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if epoch % 10 == 0:
        print(f"Epoch {epoch}, Loss: {loss.item()}")

Step 5: Deploy Model on SageMaker

import sagemaker
from sagemaker.pytorch import PyTorchModel

# Define the model location (after training, upload to S3)
model_artifact = "s3://my-bucket/gnn-model.tar.gz"

# Create a SageMaker PyTorch Model
gnn_model = PyTorchModel(
    model_data=model_artifact,
    role="AWS_IAM_ROLE",
    framework_version="1.8.1",
    py_version="py3"
)

# Deploy the model to a SageMaker Endpoint
predictor = gnn_model.deploy(instance_type="ml.m5.large", initial_instance_count=1)

Key Takeaways

✅ GNNs are essential for graph-structured data in AI applications.

✅ AWS provides a scalable ecosystem (Neptune, SageMaker, OpenSearch) for Graph ML.

✅ DGL + SageMaker enables powerful GNN training & deployment for real-world applications.

This is just the beginning—in later sections, we can explore advanced GNN architectures, real-world use cases, and cost optimization strategies on AWS! 🚀