Forem: Lam Bùi

Building an Intelligent Document Processing Pipeline with AWS Lambda and Bedrock Data Automation

Lam Bùi — Sun, 12 Oct 2025 19:22:34 +0000

Introduction

This guide provides detailed instructions on building a serverless document processing pipeline using AWS Lambda and Amazon Bedrock Data Automation (BDA). We will create an automated document processing system where Lambda functions are triggered when documents are uploaded to S3, using BDA to extract structured information from documents.

What You Will Learn

After completing this guide, you will be able to:

Create and configure Lambda functions through AWS Console
Integrate Lambda with Amazon Bedrock Data Automation
Set up S3 Event triggers for Lambda
Automatically process documents and save results
Monitor and troubleshoot Lambda functions

Solution Architecture

Knowledge Requirements

Basic understanding of AWS Console
Basic Python knowledge
Understanding of S3 and Lambda (basic level)

Article Structure

This article is divided into 5 main parts:

Part	Content
Part 1	Environment Setup (Enable Models, S3, IAM, BDA Project)
Part 2	Create and Configure Lambda Function
Part 3	Set up S3 Trigger
Part 4	Testing and Deployment
Part 5	Real-world Examples and Best Practices

Prerequisites

Before starting, ensure you have:

AWS Account with IAM permissions for Bedrock, Lambda, S3, IAM
Access to AWS Console
Selected a region supporting Amazon Bedrock Data Automation (recommended: us-east-1 or us-west-2)
Modern browser (Chrome, Firefox, Safari, or Edge)

Note about Region: Amazon Bedrock Data Automation is not available in all regions. Check Bedrock endpoints and quotas to see supported regions.

Lambda Function Code Reference

Before setting up the environment, here is the complete Lambda function code that you will use later in Part 2. Having this code available upfront allows you to understand what we're building toward.

File: lambda_function.py

import json
import boto3
import os
import time
from datetime import datetime
from urllib.parse import unquote_plus

# Initialize AWS clients
s3_client = boto3.client('s3')
bda_runtime_client = boto3.client('bedrock-data-automation-runtime')
sts_client = boto3.client('sts')

# Get configuration from environment variables
BDA_PROJECT_ARN = os.environ['BDA_PROJECT_ARN']
OUTPUT_BUCKET = os.environ['OUTPUT_BUCKET']

# Get region and account ID dynamically (Lambda provides these automatically)
AWS_REGION = os.environ['AWS_REGION']  # Lambda provides this automatically
account_id = sts_client.get_caller_identity()['Account']

# Construct BDA Profile ARN dynamically
BDA_PROFILE_ARN = f"arn:aws:bedrock:{AWS_REGION}:{account_id}:data-automation-profile/us.data-automation-v1"

def lambda_handler(event, context):
    """
    Lambda handler function to process documents with BDA.
    Triggered by S3 upload events.
    """

    try:
        # Extract S3 bucket and key from event
        s3_event = event['Records'][0]['s3']
        input_bucket = s3_event['bucket']['name'].lower()  # Ensure lowercase
        # URL decode the key (S3 event has URL-encoded keys)
        document_key = unquote_plus(s3_event['object']['key'])

        print(f"Processing document: s3://{input_bucket}/{document_key}")

        # Validate bucket name format (BDA requirement)
        if not input_bucket.replace('-', '').replace('.', '').isalnum():
            raise ValueError(f"Invalid bucket name format: {input_bucket}. Must contain only lowercase letters, numbers, dots, and hyphens.")

        # Construct S3 URIs
        input_s3_uri = f"s3://{input_bucket}/{document_key}"

        # Generate output path with timestamp
        timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
        file_name = document_key.split('/')[-1]
        output_prefix = f"processed/{timestamp}_{file_name}"
        output_s3_uri = f"s3://{OUTPUT_BUCKET}/{output_prefix}"

        # Invoke BDA to process the document
        print("Invoking Bedrock Data Automation...")
        response = bda_runtime_client.invoke_data_automation_async(
            inputConfiguration={
                's3Uri': input_s3_uri
            },
            outputConfiguration={
                's3Uri': output_s3_uri
            },
            dataAutomationConfiguration={
                'dataAutomationProjectArn': BDA_PROJECT_ARN,
                'stage': 'LIVE'
            },
            dataAutomationProfileArn=BDA_PROFILE_ARN
        )

        invocation_arn = response['invocationArn']
        print(f"BDA Invocation ARN: {invocation_arn}")

        # Wait for processing to complete (with timeout)
        max_wait_time = 240  # 4 minutes (leave 1 min for cleanup)
        wait_interval = 10
        elapsed_time = 0

        while elapsed_time < max_wait_time:
            status_response = bda_runtime_client.get_data_automation_status(
                invocationArn=invocation_arn
            )

            status = status_response['status']
            print(f"Current status: {status}")

            if status == 'Success':
                result_s3_uri = status_response['outputConfiguration']['s3Uri']
                print(f"Processing completed successfully!")
                print(f"Results saved to: {result_s3_uri}")

                return {
                    'statusCode': 200,
                    'body': json.dumps({
                        'message': 'Document processed successfully',
                        'input': input_s3_uri,
                        'output': result_s3_uri,
                        'invocationArn': invocation_arn
                    })
                }

            elif status in ['ClientError', 'ServiceError']:
                error_msg = f"Processing failed with status: {status}"
                print(error_msg)
                return {
                    'statusCode': 500,
                    'body': json.dumps({
                        'error': error_msg,
                        'invocationArn': invocation_arn
                    })
                }

            # Wait before checking again
            time.sleep(wait_interval)
            elapsed_time += wait_interval

        # Timeout reached
        print(f"Processing still in progress after {max_wait_time} seconds")
        return {
            'statusCode': 202,
            'body': json.dumps({
                'message': 'Processing initiated but not completed within Lambda timeout',
                'invocationArn': invocation_arn,
                'note': 'Check BDA console for final status'
            })
        }

    except Exception as e:
        error_msg = f"Error processing document: {str(e)}"
        print(error_msg)
        return {
            'statusCode': 500,
            'body': json.dumps({
                'error': error_msg
            })
        }

Key Features of This Code:

Automatically converts bucket names to lowercase (BDA requirement)
URL decodes file names from S3 events
Validates bucket name format
Dynamically constructs BDA Profile ARN
Polls for completion status with timeout
Comprehensive error handling and logging

When you reach Step 2.3 in Part 2, simply copy this code into your Lambda function.

Introduction to Amazon Bedrock Data Automation

Intelligent Document Processing with Generative AI

Generative AI not only drives innovation through ideation and content creation but also optimizes operational processes and increases productivity across various domains. Amazon Bedrock Data Automation (BDA) is a fully managed service that provides Intelligent Document Processing (IDP) capabilities enhanced by generative AI.

Why Choose BDA?

1. Complete Automation

Enterprises can extract significant value from IDP enhanced with generative AI. By integrating generative AI capabilities into IDP solutions, organizations can:

Advanced Document Understanding: Deep analysis of structure and semantics
Structured Data Extraction: Automatic transformation from unstructured to structured data
Automatic Classification: Document type recognition and appropriate routing
Information Retrieval: Search and retrieve information from unstructured text

2. BDA vs Direct Bedrock API

Aspect	Direct Bedrock API	BDA (Managed Service)
Complexity	Need to build pipeline	Fully managed, integrated
Document Understanding	Requires prompt engineering	Built-in document intelligence
Multimodal	Only processes images	Supports various formats: PDF, images, audio, video
Output Format	Custom JSON	Standardized JSON + Markdown/HTML/CSV
Bounding Boxes	Not available	Automatically provides element positions
Scalability	Self-managed	Auto-scaling
Best For	Custom IDP solutions	Enterprise document processing

Business Value and Use Cases

Government & Public Sector

Process and extract data from:

Birth certificate applications, ID/Passport applications
Immigration and visa records
Legal contracts and government forms

Benefits: Reduce processing time from days to minutes, improve citizen service quality.

Healthcare

Extract and organize information from:

Electronic medical records
Insurance claim requests
Prescriptions and test results
Clinical trial records

Benefits: Improve data accuracy, increase information accessibility for better patient care.

Finance & Banking

Automate processing:

Loan and credit applications
Financial reports and tax documents
Contracts and agreements
Compliance documentation

Benefits: Reduce manual work, increase operational efficiency, reduce compliance risks.

Logistics & Supply Chain

Process:

Shipping documents and invoices
Purchase orders and warehouse receipts
Supplier contracts
Customs certificates

Benefits: Optimize processes, increase supply chain visibility.

Retail & E-commerce

Automate:

Customer orders
Product catalogs and descriptions
Invoices and receipts
Marketing documents

Benefits: Personalize customer experience, process orders quickly and efficiently.

Why Combine BDA with Lambda?

Combining BDA with AWS Lambda creates a powerful serverless IDP pipeline:

Event-Driven: Automatically processes when new documents arrive
Scalable: Automatically scales with document volume
Cost-Effective: Pay only when processing
Low Maintenance: No server management needed
Integration Ready: Easy to integrate with existing systems

Statistics and Impact

According to AWS, organizations deploying IDP with generative AI have achieved:

90%+ reduction in document processing time
60-70% cost savings in operations
95%+ accuracy in data extraction
80% reduction in manual intervention needs

AWS Services in This Guide

This guide uses the following AWS services to build a serverless IDP pipeline:

Amazon Bedrock Data Automation

Fully managed service providing intelligent document processing capabilities with generative AI. BDA automatically processes documents and extracts structured information without needing to orchestrate complex tasks.

Key Features:

Document classification and extraction
Multi-granularity analysis (document, page, element level)
Generative summaries and descriptions
Support for diverse formats: PDF, images, audio, video

AWS Lambda

Serverless computing service that allows running code without managing servers. Lambda automatically scales and you only pay for compute time used.

Role in solution:

Triggered when new document uploaded
Calls BDA API to process document
Handles and saves results

Amazon S3 (Simple Storage Service)

Highly scalable, durable, and secure object storage service.

Role in solution:

Store input documents
Store output results from BDA
Trigger S3 events for Lambda

Amazon CloudWatch

Monitoring and observability service for AWS resources and applications.

Role in solution:

Collect logs from Lambda execution
Monitor metrics (invocations, errors, duration)
Troubleshooting and debugging

AWS IAM (Identity and Access Management)

Securely manage access and permissions for AWS resources.

Role in solution:

Create execution role for Lambda
Grant permissions to S3, Bedrock, CloudWatch
Security and access control

Detailed Workflow

1. USER uploads document
   │
   ▼
2. S3 Event Notification
   │
   ▼
3. Lambda Function triggered
   │
   ├─▶ Read document from S3
   │
   ├─▶ Call BDA InvokeDataAutomationAsync API
   │   │
   │   ▼
   │   BDA Processing:
   │   ├─ Document ingestion
   │   ├─ Structure analysis
   │   ├─ Content extraction (text, tables, figures)
   │   ├─ Semantic enrichment (AI summaries)
   │   └─ Result formation (JSON, Markdown, CSV)
   │
   ├─▶ Poll for completion status
   │
   └─▶ Save results to S3 output bucket

Difference from Traditional IDP

Traditional IDP	BDA-Powered IDP
Rule-based extraction	AI-powered understanding
Template dependency	Template-free processing
Manual training needed	Pre-trained models
Limited format support	Multi-format support
No semantic understanding	Deep semantic analysis
Fixed output structure	Flexible, rich output

Part 1: Environment Setup

Step 1.1: Enable Model Access in Amazon Bedrock

Purpose: Enable access to foundation models required for BDA.

Important: This is a mandatory step before using Amazon Bedrock Data Automation. Without enabling model access, BDA cannot process documents.

Models to Enable:

We need to enable the following models for this guide:

Amazon Models: All models
Claude Models:
- Claude 3.5 Haiku
- Claude 3 Sonnet
- Claude 3.5 Sonnet
Cohere Models:
- Cohere Rerank 3.5

Implementation Steps:

Open Amazon Bedrock Console
- Search for Bedrock in the top search bar
- Click on Amazon Bedrock
Access Model Access
- In the left menu (or navigation menu), click on Model access
- Click Modify model access button (or Request model access if first time)
Select Models
In the models list, select the following:
Amazon Models:
- Check all Amazon models Claude Models (by Anthropic):
- Check Claude 3.5 Haiku
- Check Claude 3 Sonnet
- Check Claude 3.5 Sonnet Cohere Models:
- Check Cohere Rerank 3.5
Submit Request
- Click Next button at bottom right
- Review selected models
- Click Submit to request access
Wait for Approval
- Most models will be approved instantly (status: Access granted)
- Some models may take a few minutes for provisioning
- Refresh page to see status updates

When all models show status "Access granted", you are ready to continue.

Step 1.2: Create S3 Buckets

Purpose: Create S3 buckets to store input documents and output results.

Log in to AWS Management Console
Search for and select S3 service
Click Create bucket button

Create Input Bucket:

Enter information:
- Bucket name: bda-workshop-input-demo-xyz789 (replace xyz789 with your random string to ensure bucket name is unique)
- Block Public Access settings: Keep default (Block all public access)
Scroll down and click Create bucket

Create Output Bucket:
Repeat steps 3-5 with:
- Bucket name: bda-workshop-output-demo-xyz789 (use same suffix as input bucket)
- Other configurations keep default

Step 1.3: Create IAM Role for Lambda

Purpose: Create IAM role with sufficient permissions for Lambda to access S3 and Bedrock.

In AWS Console, search for and select IAM
In left menu, select Roles
Click Create role button

Select Trusted Entity:
Trusted entity type: Select AWS service
Use case: Select Lambda
Click Next

Assign Permissions:
In policy search box, find and select the following policies (check the checkbox):
- AWSLambdaBasicExecutionRole (for CloudWatch Logs)
- AmazonS3FullAccess (to read/write S3)
- AmazonBedrockFullAccess (to use Bedrock BDA) Security Note: In production environments, create custom policies with minimum necessary permissions instead of using FullAccess policies.
Click Next

Name and Create Role:

Role name: BDA-Lambda-ExecutionRole
Description: `Execution role for BDA document processing Lambda function
Scroll down and click Create role ---

Step 1.4: Create BDA Project

Purpose: Create BDA project to configure document processing.

In AWS Console, search for and select Amazon Bedrock
In left menu, select Data Automation > Set-up Project
Click Create project button

Configure Project:

Enter information:
- Project name: document-processing-project
- Select ** Create Project**
In Standard output configuration section, configure:
- Click Edit
- Granularity types:
  - Check DOCUMENT
  - Check PAGE
  - Check ELEMENT
- Bounding box: Select ENABLED
- Generative field: Select ENABLED
- Text format types:
  - Check MARKDOWN
- Additional file format: Select ENABLED
Click Save changes
Wait for project status to change to Changes saved successfully.
Copy and save Project ARN (will be used in Lambda configuration step)
- Click on project name to view details
- Copy ARN from project details page
- Example ARN: arn:aws:bedrock:us-east-1:111111111111:data-automation-project/abc123xyz

Part 2: Create Lambda Function

Step 2.1: Create Lambda Function

In AWS Console, search for and select Lambda
Click Create function button
Select Author from scratch
Enter information:
- Function name: BDA-Document-Processor
- Runtime: Select Python 3.12
- Architecture: Select x86_64 #### Permissions:
In Permissions section, select Use an existing role
Existing role: Select BDA-Lambda-ExecutionRole (role created in step 1.3)
Click Create function

Step 2.2: Configure Lambda Function

Increase Timeout and Memory:

In Lambda function page, select Configuration tab
Select General configuration > Click Edit
Configure:
- Memory: 512 MB
- Timeout: 5 min (5 minutes)
- Description: Processes documents using Bedrock Data Automation
Click Save
Still in Configuration tab, select Environment variables
Click Edit > Click Add environment variable
Add the following variables:

Key	Value	Description
`BDA_PROJECT_ARN`	Paste ARN saved in step 1.4	ARN of created BDA project
`OUTPUT_BUCKET`	`bda-workshop-output-demo-xyz789`	Output bucket name (replace with your bucket name)

8.Click Save

Step 2.3: Write Lambda Code

Return to Code tab
In editor, delete sample code and replace with the complete Lambda function code provided in the Lambda Function Code Reference section above (before Part 1).
Click Deploy to save code

The code handles S3 events, validates bucket names, invokes BDA, and polls for completion status with comprehensive error handling.

Part 3: Configure S3 Trigger

Step 3.1: Add S3 Trigger for Lambda

Purpose: Configure Lambda to automatically run when new files are uploaded to S3.

In Lambda function page, select Configuration tab
Select Triggers in left menu
Click Add trigger
Select a source: Select S3
Bucket: Select bda-workshop-input-demo-xyz789 (input bucket created in step 1.2)
Event type: Select All object create events
Prefix (optional): Leave blank (process all files) or enter documents/ if only want to process files in documents folder
Suffix (optional): Enter .pdf if only want to process PDF files
Check the checkbox I acknowledge that using the same S3 bucket...
Click Add

Part 4: Testing and Deployment

Step 4.1: Prepare Test Document

Download a PDF sample file to your computer (or use existing file)

Important - File Naming:

File names should only contain: letters (a-z, A-Z), numbers (0-9), hyphens (-), underscores (_), dots (.)
Should NOT use: spaces, special characters, Vietnamese characters
Good file name examples: document-test.pdf, report_2024.pdf
File names to avoid: report document.pdf, document (1).pdf

Step 4.2: Upload and Test

Open S3 Console
Go to bucket bda-workshop-input-demo-xyz789 (your input bucket)
Click Upload
Click Add files and select test PDF file
Click Upload

Step 4.3: Check Lambda Execution

Return to Lambda Console
Select function BDA-Document-Processor
Select Monitor tab
Click View CloudWatch logs
Click on latest log stream (most recent time)
View logs to verify:
- Lambda was triggered
- Document being processed
- BDA invocation successful
- Processing completed

Step 4.4: Check Output

Open S3 Console
Go to bucket bda-workshop-output-demo-xyz789 (your output bucket)
Go to processed/ folder
You will see folder with timestamp and file name
Inside that folder will have:
- job_metadata.json - job metadata
- standard-output.json - main processing results
- Other files like markdown, CSV (if any)
Download standard-output.json and view results:
- Document summary
- Extracted tables
- Figures
- Page-level information
- Element details

Real-World Example: Government Birth Certificate Processing

Use Case from AWS Blog

A real-world example from AWS Machine Learning Blog illustrates how IDP with generative AI solves real problems:

The Problem

Government agency issuing birth certificates receives applications through multiple channels:

Online applications
Forms completed at physical locations
Mailed paper applications

Current Manual Process:

Scan paper applications
Staff manually read and enter into system
Check and validation
Save to database

Issues:

Very time-consuming
High labor costs
Prone to manual data entry errors
Not scalable when volume increases
Complex if forms are in multiple languages (English, Spanish, etc.)

Solution with BDA

With architecture similar to this guide, but with additions:

SQS Queue: Buffer to process messages reliably
DynamoDB: Store extracted data
Multi-language Support: Automatically translate and extract

Upload Form → S3 → Lambda → BDA → SQS → Lambda → DynamoDB ↓ Auto-detect language Extract all fields Translate if needed

Results Achieved

Before:

15-20 minutes/application (manual)
High labor costs
85-90% accuracy (human error)

After (with BDA):

< 1 minute/application (automated)
60-70% cost savings
95%+ accuracy
Process multiple languages
Scale to thousands of applications/day

Fields Extracted Automatically

BDA can extract complex information:

Applicant information (name, address, contact)
Birth certificate recipient information
Parent information
Fee payment information
Signatures and dates
Bonus: Automatically translate from Spanish to English

Extending the Solution

You can extend this solution in the following directions:

Add SQS Queue: Buffer processing and retry logic
Add DynamoDB: Store structured data
Custom Extraction: Define fields to extract for specific domain
Multi-language: Process documents in multiple languages
Human-in-the-loop: Validation for critical data

Cleanup Resources (Optional)

If you do not want to continue using and avoid incurring costs:

Delete Lambda Function:

Lambda Console > Select function > Actions > Delete

Delete S3 Buckets:

S3 Console > Select bucket > Empty (delete all objects)
Then select bucket > Delete

Delete IAM Role:

IAM Console > Roles > Select role > Delete

Delete BDA Project:

Bedrock Console > Data Automation > Projects > Select project > Delete

Reference Documentation

AWS Documentation:

AWS Lambda Developer Guide - Complete Lambda guide
Amazon Bedrock Data Automation - Official BDA documentation
Amazon Bedrock User Guide - Bedrock overview
S3 Event Notifications - S3 trigger configuration
Standard Output in BDA - Output configuration details

API References:

AWS Blogs and Case Studies:

Conclusion

Congratulations! You have successfully completed and created a serverless document processing pipeline with AWS Lambda and Amazon Bedrock Data Automation!

Summary of What You Learned

In this article, we covered:

Understanding BDA and IDP: Learned about Intelligent Document Processing with generative AI and benefits of Amazon Bedrock Data Automation

Building Infrastructure: Created S3 buckets, IAM roles, and BDA project through AWS Console

Developing Lambda Function: Wrote Python code to integrate with BDA API

Event-Driven Architecture: Set up S3 event triggers for automated processing

Testing and Monitoring: Deployed, tested, and monitored solution with CloudWatch

Troubleshooting: Handled common issues and best practices

Next Steps

Now that you have a solid foundation, continue exploring:

Expand to multimodal: Try processing images, audio, and video with BDA
Integrate downstream systems: Connect with DynamoDB, API Gateway, or business applications
Advanced patterns: Implement human-in-the-loop workflows, model evaluation
Production readiness: Error handling, DLQ, cost optimization, security hardening
Explore RAG: Build multimodal RAG applications with BDA and Knowledge Bases

Acknowledgments

Thank you for reading this article! Hope this guide helps you start your journey with Amazon Bedrock Data Automation. Wish you success in building intelligent document processing solutions!

Building an Intelligent Document Processing Pipeline on AWS with S3, Textract, Comprehend, and DynamoDB

Lam Bùi — Sat, 04 Oct 2025 15:48:43 +0000

Introduction

In today's digital world, organizations are drowning in unstructured data. Documents, images, and PDFs contain valuable information that often remains untapped due to the manual effort required to extract and analyze it. What if we could automatically process these documents, extract meaningful insights, and store structured data for further analysis?

This blog post will guide you through building a complete intelligent document processing pipeline using AWS services. Our pipeline will automatically:

Extract text from images and PDFs using Amazon Textract
Analyze content for entities, key phrases, and sentiment using Amazon Comprehend
Store structured results in DynamoDB for easy querying and analysis
Process documents automatically when uploaded to S3

What We're Building

Our pipeline creates a seamless flow where:

Documents are uploaded to an S3 bucket (images, PDFs, etc.)
Lambda function triggers automatically when new files arrive
Textract extracts text and identifies document layout
Comprehend analyzes the extracted text for insights
Results are stored in DynamoDB with structured metadata

Architecture Overview

System Requirements

Lambda Runtime: Python 3.10
Memory: 1024 MB (recommended)
Timeout: 120 seconds
Environment Variables:
- DDB_TABLE: SmartDocResults (default)
- LANG: en (default)

Step-by-Step Implementation

Step 1: Create DynamoDB Table

Navigate to AWS Console → DynamoDB
Click "Create table"
Configure:
- Table name: SmartDocResults
- Partition key: doc_id (String)
- Sort key: paragraph_id (String)
Click "Create table"
Wait for table status = "Active"

Step 2: Create S3 Bucket

AWS Console → S3
Click "Create bucket"
Configure:
- Bucket name: your-smart-doc-bucket (change to unique name)
- Region: Choose your preferred region
Click "Create bucket"
Remember the bucket name for IAM policy

Step 3: Create IAM Policy

AWS Console → IAM → Policies
Click "Create policy"
Switch to "JSON" tab
Copy content from iam_policy.json and replace placeholders:
- ACCOUNT_ID: Your AWS account ID
- REGION: Your region (e.g., us-east-1)
- BUCKET_NAME: S3 bucket name from step 2
Click "Next: Tags" → "Next: Review"
Name the policy: SmartDocLambdaPolicy
Click "Create policy"

Least-privilege IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3Access",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::BUCKET_NAME/*"
        },
        {
            "Sid": "TextractAccess",
            "Effect": "Allow",
            "Action": [
                "textract:AnalyzeDocument"
            ],
            "Resource": "*"
        },
        {
            "Sid": "ComprehendAccess",
            "Effect": "Allow",
            "Action": [
                "comprehend:DetectEntities",
                "comprehend:DetectKeyPhrases",
                "comprehend:DetectSentiment"
            ],
            "Resource": "*"
        },
        {
            "Sid": "DynamoDBAccess",
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:GetItem",
                "dynamodb:Query",
                "dynamodb:Scan"
            ],
            "Resource": "arn:aws:dynamodb:REGION:ACCOUNT_ID:table/SmartDocResults"
        },
        {
            "Sid": "CloudWatchLogs",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:REGION:ACCOUNT_ID:*"
        }
    ]
}

Step 4: Create IAM Role for Lambda

AWS Console → IAM → Roles
Click "Create role"
Select "AWS service" → "Lambda"
Click "Next"
In "Permissions" tab:
- Find and select the SmartDocLambdaPolicy you just created
- Check the policy
Click "Next: Tags"
Name the role: SmartDocLambdaRole
Click "Create role"

Step 5: Create Lambda Function

AWS Console → Lambda → Functions
Click "Create function"
Select "Author from scratch"
Configure:
- Function name: SmartDocProcessor
- Runtime: Python 3.10
- Architecture: x86_64
- Change default execution role: Select "Use an existing role" → SmartDocLambdaRole
Click "Create function"

Step 6: Configure Lambda Function

In Lambda function, scroll to "Code" section
Delete default code and paste content from lambda_function.py
Click "Deploy"
Configure "Configuration":
- General:
  - Memory: 1024 MB
  - Timeout: 2 minutes
- Environment variables:
  - DDB_TABLE: SmartDocResults
  - LANG: en

import json
import boto3
import os
from datetime import datetime
from urllib.parse import unquote_plus
from typing import List, Dict, Any, Optional
import logging
from decimal import Decimal

# Configure logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

# Initialize AWS clients
textract = boto3.client('textract')
comprehend = boto3.client('comprehend')
dynamodb = boto3.resource('dynamodb')

# Environment variables
DDB_TABLE = os.environ.get('DDB_TABLE', 'SmartDocResults')
LANG = os.environ.get('LANG', 'en')

def lambda_handler(event, context):
    """
    Main Lambda handler for S3 → Textract → Comprehend → DynamoDB pipeline
    """
    logger.info(f"Processing event: {json.dumps(event)}")

    # Get DynamoDB table
    table = dynamodb.Table(DDB_TABLE)

    # Process each S3 record
    for record in event.get('Records', []):
        try:
            # Extract S3 information
            bucket = record['s3']['bucket']['name']
            key = unquote_plus(record['s3']['object']['key'])
            doc_id = os.path.basename(key)

            logger.info(f"Processing document: {doc_id} from bucket: {bucket}")

            # Step 1: Extract text using Textract
            text_lines = extract_text_from_s3(bucket, key)
            if not text_lines:
                logger.warning(f"No text extracted from {doc_id}")
                continue

            # Step 2: Split into paragraphs
            paragraphs = split_paragraphs(text_lines)
            logger.info(f"Found {len(paragraphs)} paragraphs in {doc_id}")

            # Step 3: Process each paragraph with Comprehend
            for paragraph_id, paragraph in enumerate(paragraphs, 1):
                if len(paragraph) >= 20:  # Only process paragraphs with >= 20 characters
                    logger.info(f"Processing paragraph {paragraph_id} (length: {len(paragraph)})")

                    # Analyze with Comprehend
                    entities = detect_entities_safe(paragraph, LANG)
                    key_phrases = detect_key_phrases_safe(paragraph, LANG)
                    sentiment = safe_detect_sentiment(paragraph, LANG)

                    # Convert float values to Decimal for DynamoDB
                    entities = convert_floats_to_decimal(entities)
                    key_phrases = convert_floats_to_decimal(key_phrases)

                    # Save to DynamoDB
                    item = {
                        'doc_id': doc_id,
                        'paragraph_id': str(paragraph_id),  # Convert to string
                        'content': paragraph,
                        'entities': entities,
                        'key_phrases': key_phrases,
                        'sentiment': sentiment,
                        'created_at': datetime.utcnow().isoformat() + 'Z'
                    }

                    table.put_item(Item=item)
                    logger.info(f"Saved paragraph {paragraph_id} to DynamoDB")
                else:
                    logger.info(f"Skipping paragraph {paragraph_id} (too short: {len(paragraph)} chars)")

            logger.info(f"Successfully processed document: {doc_id}")

        except Exception as e:
            logger.error(f"Error processing record {record}: {str(e)}")
            # Continue processing other records
            continue

    return {
        'statusCode': 200,
        'body': json.dumps('Processing completed')
    }

def extract_text_from_s3(bucket: str, key: str) -> List[str]:
    """
    Extract text from S3 object using Textract synchronous API
    """
    try:
        response = textract.analyze_document(
            Document={
                'S3Object': {
                    'Bucket': bucket,
                    'Name': key
                }
            },
            FeatureTypes=['LAYOUT']
        )

        # Extract LINE blocks and sort by reading order
        lines = []
        line_blocks = [block for block in response['Blocks'] if block['BlockType'] == 'LINE']

        # Sort by reading order (top to bottom, left to right)
        line_blocks.sort(key=lambda x: (x['Geometry']['BoundingBox']['Top'], 
                                      x['Geometry']['BoundingBox']['Left']))

        for block in line_blocks:
            if 'Text' in block:
                lines.append(block['Text'])

        logger.info(f"Extracted {len(lines)} lines from document")
        return lines

    except Exception as e:
        logger.error(f"Error extracting text from {bucket}/{key}: {str(e)}")
        return []

def split_paragraphs(lines: List[str]) -> List[str]:
    """
    Split lines into paragraphs based on spacing and punctuation rules
    """
    if not lines:
        return []

    paragraphs = []
    current_paragraph = []

    for line in lines:
        line = line.strip()
        if not line:
            # Empty line - end current paragraph
            if current_paragraph:
                paragraphs.append(' '.join(current_paragraph))
                current_paragraph = []
        elif line.endswith('.') and len(line) > 1:
            # Line ends with period - add to current paragraph and end it
            current_paragraph.append(line)
            paragraphs.append(' '.join(current_paragraph))
            current_paragraph = []
        else:
            # Regular line - add to current paragraph
            current_paragraph.append(line)

    # Add final paragraph if exists
    if current_paragraph:
        paragraphs.append(' '.join(current_paragraph))

    return paragraphs

def detect_entities_safe(text: str, language_code: str) -> List[Dict[str, Any]]:
    """
    Safely detect entities using Comprehend with error handling
    """
    try:
        response = comprehend.detect_entities(
            Text=text,
            LanguageCode=language_code
        )
        return response['Entities']
    except Exception as e:
        logger.error(f"Error detecting entities: {str(e)}")
        return []

def detect_key_phrases_safe(text: str, language_code: str) -> List[Dict[str, Any]]:
    """
    Safely detect key phrases using Comprehend with error handling
    """
    try:
        response = comprehend.detect_key_phrases(
            Text=text,
            LanguageCode=language_code
        )
        return response['KeyPhrases']
    except Exception as e:
        logger.error(f"Error detecting key phrases: {str(e)}")
        return []

def safe_detect_sentiment(text: str, language_code: str) -> str:
    """
    Safely detect sentiment with chunking for large texts
    Comprehend has a 5000 byte limit for DetectSentiment
    """
    try:
        # Check if text is small enough for single call
        if len(text.encode('utf-8')) <= 4500:
            response = comprehend.detect_sentiment(
                Text=text,
                LanguageCode=language_code
            )
            return response['Sentiment']

        # For large texts, chunk and aggregate
        logger.info(f"Text too large for single sentiment call, chunking...")
        chunks = chunk_text(text, 4000)  # Leave some buffer
        sentiments = []

        for chunk in chunks:
            try:
                response = comprehend.detect_sentiment(
                    Text=chunk,
                    LanguageCode=language_code
                )
                sentiments.append(response['Sentiment'])
            except Exception as e:
                logger.error(f"Error detecting sentiment for chunk: {str(e)}")
                sentiments.append('UNKNOWN')

        # Aggregate sentiments (simple majority vote)
        return aggregate_sentiments(sentiments)

    except Exception as e:
        logger.error(f"Error detecting sentiment: {str(e)}")
        return 'UNKNOWN'

def chunk_text(text: str, max_bytes: int) -> List[str]:
    """
    Split text into chunks that don't exceed max_bytes
    """
    chunks = []
    current_chunk = ""

    for word in text.split():
        test_chunk = current_chunk + " " + word if current_chunk else word

        if len(test_chunk.encode('utf-8')) <= max_bytes:
            current_chunk = test_chunk
        else:
            if current_chunk:
                chunks.append(current_chunk)
            current_chunk = word

    if current_chunk:
        chunks.append(current_chunk)

    return chunks

def aggregate_sentiments(sentiments: List[str]) -> str:
    """
    Aggregate multiple sentiment results into a single sentiment
    """
    if not sentiments:
        return 'UNKNOWN'

    # Count sentiment occurrences
    sentiment_counts = {}
    for sentiment in sentiments:
        sentiment_counts[sentiment] = sentiment_counts.get(sentiment, 0) + 1

    # Return the most common sentiment
    return max(sentiment_counts, key=sentiment_counts.get)

def convert_floats_to_decimal(data: Any) -> Any:
    """
    Recursively convert float values to Decimal for DynamoDB compatibility
    """
    if isinstance(data, dict):
        return {key: convert_floats_to_decimal(value) for key, value in data.items()}
    elif isinstance(data, list):
        return [convert_floats_to_decimal(item) for item in data]
    elif isinstance(data, float):
        return Decimal(str(data))
    else:
        return data

Step 7: Add S3 Trigger

In Lambda function → "Add trigger"
Select "S3"
Configure:
- Bucket: Select bucket created in step 2
- Event type: All object create events
- Prefix: (leave empty or add folder path)
- Suffix: (leave empty)
Check "Recursive invocation" if you want to process files in subfolders
Click "Add"

Step 8: Test with Sample File

Upload a test file to your S3 bucket:
- Supported file types: JPG, PNG, PDF (1 page)
- Size: < 10MB (Textract sync limit)
Check CloudWatch Logs:
- Lambda function → "Monitor" → "View CloudWatch logs"
- Look for log group: /aws/lambda/SmartDocProcessor
Check DynamoDB:
- DynamoDB → Tables → SmartDocResults → "Items"
- View created items ### Step 9: Verify Results

Sample DynamoDB Item

{
  "doc_id": "sample-document.pdf",
  "paragraph_id": "1",
  "content": "This is the first paragraph of the document with important information.",
  "entities": [
    {
      "Text": "sample-document.pdf",
      "Type": "OTHER",
      "Score": 0.99
    }
  ],
  "key_phrases": [
    {
      "Text": "important information",
      "Score": 0.99
    }
  ],
  "sentiment": "POSITIVE",
  "created_at": "2024-01-15T10:30:00.000Z"
}

How to Verify

CloudWatch Logs: View logs for debugging and monitoring processing
DynamoDB Console: View items with complete information
S3 Console: Confirm file was uploaded

Troubleshooting

Common Issues:

Permission denied: Check IAM role has sufficient permissions
Textract error: File too large or unsupported format
DynamoDB error: Check table name and permissions
Timeout: Increase memory or timeout for Lambda
Comprehend error: Check text language and size limits

Debug Tips:

View CloudWatch Logs for specific errors
Verify IAM policy has correct placeholders
Confirm S3 bucket and DynamoDB table exist
Test Textract and Comprehend services separately

Next Steps & Advanced Features

1. Multi-page PDF Processing

Use Textract async API (StartDocumentAnalysis)
Combine with SNS for completion notifications
Modify Lambda to handle SNS events

2. Custom Comprehend Models

Create custom entity recognizers for specific domains
Use custom classification models
Improve analysis accuracy

3. Security Enhancements

Use KMS for DynamoDB data encryption
Enable S3 server-side encryption
Deploy Lambda in VPC for high security
Use IAM roles instead of access keys

4. Monitoring & Alerting

Set up CloudWatch alarms
Create dashboards to monitor pipeline
Use X-Ray for request tracing

Conclusion

You've successfully built an intelligent document processing pipeline that can automatically extract and analyze content from various document types. This solution provides:

Automated processing of uploaded documents
Intelligent text extraction using AWS Textract
Content analysis with entity recognition, key phrase extraction, and sentiment analysis
Structured data storage in DynamoDB for easy querying
Scalable architecture that can handle varying workloads

This pipeline can be extended for various use cases such as:

Document classification and routing
Content moderation
Data extraction for business intelligence
Automated compliance checking
Customer support ticket analysis

The combination of AWS services provides a robust, scalable solution that can grow with your needs while maintaining security and cost-effectiveness.

Modernize Mainframe Applications with AWS Transform

Lam Bùi — Tue, 23 Sep 2025 02:51:25 +0000

I. Concepts

1. What is AWS Transform?

AWS Transform is a new AWS service that uses agentic AI to automate modernization and migration of infrastructure, applications, and source code to AWS. It reduces the time, effort, and risk of handling legacy or complex systems such as mainframes, VMware, and .NET workloads by analyzing, refactoring, and redeploying on cloud.

2. Primary Use Cases

Mainframe Modernization

Transform COBOL, JCL, CICS, Db2, VSAM into modern Java applications.
Automatically analyze code, extract business logic, generate technical documents, and refactor.

VMware Migration

Migrate virtual machines from on‑premises VMware to Amazon EC2.
Auto-generate migration plans, configure VPC networking, and convert VMs to EC2.

.NET Modernization

Upgrade from legacy .NET Framework to cross‑platform .NET.
Integrates with Visual Studio; automatically resolves dependencies, builds, and tests.

Migration Assessment

Pre-migration cost and feasibility assessment.
AI-driven recommendations for EC2 sizing, BYOL optimization, and cost analysis.

3. Key Capabilities

Agentic AI with human-in-the-loop (HITL) for complex tasks and decisions.
End‑to‑end automation: discovery → planning → code refactor → deployment.
Flexible collaboration: role-based access (Admin, Approver, Contributor, Reader).
Workspaces to isolate projects, with dedicated jobs and connectors.
Worklog & dashboards to track progress and decisions.

4. Typical Workflow

1) Environment setup

Create an AWS account with administrative permissions.
Enable the AWS Transform service.
Configure AWS IAM Identity Center for users and groups.

2) Create a Workspace

Dedicated per transformation project.
Contains connectors, jobs, and artifacts.

3) Create a Connector

Connect to target environments (VMware account, GitHub repository, other AWS accounts).
Configure IAM policies and encryption with KMS.

4) Create a Job

Choose job type: VMware Migration, Mainframe Modernization, .NET Transform, etc.
Upload inputs (RVTools export, source code, JCL, spreadsheets, Git repo, etc.).

5) Execute Workflow steps

VMware: Discovery → Grouping → Generate VPC → Replicate & Migrate
Mainframe: Analyze COBOL → Generate Docs → Extract Logic → Refactor → Build Java
.NET: Connect Git → Resolve Dependencies → Transform Code → Test & Build
Assessment: Upload file → Cost estimation → Migration feasibility

6) Track Progress

Worklog: detailed action log.
Dashboard: overall job status and KPIs.

5. Security & Governance

Data encrypted by default using AWS KMS.
Option to use customer‑managed KMS keys for tighter control.
Comprehensive IAM policies for access control, connector creation, and deployments.
Supports cross‑region processing and compliance validation.

6. Roles

Role	Core Permissions
Admin	Full control: create workspace, job, connector; approve HITL steps
Approver	Near‑admin; cannot modify users, connectors, or workspaces
Contributor	Execute jobs, upload files, respond to HITL; cannot approve critical steps
Reader	Read‑only access to progress and artifacts

7. Highlights by Use Case

VMware Migration

AI‑generated AWS VPC topology mirroring on‑prem networks.
Automatic wave planning from discovery data.
Import from NSX or RVTools via collector or direct file upload.
Supports Hub‑and‑Spoke and Isolated VPC topologies.

Mainframe Modernization

Full support for COBOL, JCL, Db2, and VSAM.
Clear stages: analyze → extract logic → refactor to Java.
Re‑run per stage for iterative validation.
Outputs modern Java applications deployable on AWS.

.NET Modernization

Connect GitHub repository to retrieve code.
Automatically resolve missing libraries and dependencies; refactor to .NET 6/7/8.
Integrate with Visual Studio for review and edits.
Build and deploy to test environments.

8. Core Benefits

Accelerate migration/modernization by 50–80%.
Reduce manual errors on complex workloads.
Preserve critical business logic via intelligent extraction.
Scale for large organizations: multi‑account, multi‑region, IAM governance.

II. Step‑by‑Step Guides

1) Modernize Mainframe Applications with AWS Transform

1.1 Objectives

Use AWS Transform to modernize mainframe COBOL applications (commonly on IBM z/OS) by:

Analyzing COBOL/JCL/CICS code.
Generating technical documentation and extracting business logic.
Decomposing a monolith into functional domains.
Refactoring and transforming code into modern Java.
Planning migration waves.
Creating deployment pipelines on AWS.

The journey follows four phases: Assess → Mobilize → Migrate & Modernize → Operate & Optimize.

1.2 Procedure

Step 1: Prepare Source Code

Retrieve the demo repository CardDemo: https://github.com/aws-samples/aws-mainframe-modernization-carddemo
Select Code → Download ZIP.

After extracting, locate the app/ folder with multiple subfolders.

Step 2: Keep Only Required Folders

Keep these under app/:

✅ Keep	Reason
app/cbl/	COBOL programs
app/jcl/	Job Control Language scripts
app/cpy/	Copybooks (shared data definitions)
app/bms/	CICS screen maps (if present)
app/data/	EBCDIC data (test or demo)
app/proc/	JCL procedures and helper programs

Remove:

❌ Remove	Reason
catlg/, ctl/, csd/, cpy-bms/	Not essential for core analysis

Step 3: Zip the Source

Rename folder app to mainframe-src, then zip it.

Step 4: Upload to S3

Open AWS Console → S3: https://s3.console.aws.amazon.com/s3/
Create a bucket (e.g., lam-mainframe-transform).

Upload mainframe-src.zip to the bucket.

Step 5: Create Workspace and Job

Open the AWS Transform portal and sign in via IAM Identity Center.
Open your workspace or create a new one.
Select Create a job.

Choose job type: Mainframe Modernization.

To experience the full pipeline, select: Analyze code, Generate technical documentation, Decompose code, Plan migration sequence, and Transform code to Java.
Name the job and create it.

Step 6: Start Modernization

Skip Add collaborators and proceed to Connect to AWS account.
Enter your AWS Account ID and select Next.

In S3 console, copy the Bucket ARN of lam-mainframe-transform and paste it in AWS Transform to create a connector. Select Create connector.

Copy the verification link, open it in a new tab, and approve access to S3 for AWS Transform.

After admin approval, select Send to AWS Transform to continue.

Enter the S3 URI of mainframe-src.zip and select Send to AWS Transform.

Step 7: Analyze COBOL Code

When Analyze code completes, review results and select Send to AWS Transform to proceed.

Step 8: Generate Technical Documentation

Open Summary and select all folders in Selected Files.

Select Send to AWS Transform to begin Decompose code.

When complete, view the generated documentation in: s3://lam-mainframe-transform/transform-output/

Step 9: Decompose into Domains

Create a domain classification. Each domain represents a separate functional area.
Recommended initial domains and seed files:

Domain	Seed Files	Cyclomatic Complexity	Description (EN)	Reason for Selection
Customer Domain	CBCUS01C.cbl, COUSR00C.cbl	20 / 102	Customer and user information handling	Key customer/user management, moderate complexity
Transaction Domain	CBTRN01C.cbl, COTRN00C.cbl	59 / 100	Processing and managing financial transactions	Core transaction logic, high complexity
Card Domain	COCRDLIC.cbl, COCRDUPC.cbl	144 / 159	Card data listing and updates	Primary card ops, high complexity
Account Domain	CBACT01C.cbl, CBACT04C.cbl	1 / 98	Account-level data and operations	Foundational account logic, moderate complexity
Admin Domain	COADM01C.cbl	36	Administrative features and controls	Focused module, clearly separable
Common/Utility	CBSTM03A.cbl, CSUTLDTC.cbl	1 / 11	Shared utilities and helper functions	Commonly reused lightweight utilities

In AWS Transform, choose Actions → Create domain.

Create Customer Domain with description: Handles customer and user information.

Select CBTRN01C.cbl and choose Mark as seed.

Add COUSR00C.cbl as a seed file as well. Select Create.

Select Save to finalize the domain.

Repeat for other domains listed above. For Common/Utility Domain, enable Set as a common domain.

When domains are ready, select Decompose and then Send to AWS Transform.

Step 10: Plan Migration Waves

In Plan migration wave, review the suggested references.

Select Add reference, provide referenced waves, and choose Add and regenerate.

When complete, select Send to AWS Transform to proceed to Refactor code.

Step 11: Refactor Code

In Transform, select all domains and choose Send to AWS Transform.

When refactoring completes, select View result to find generated.zip in S3.

Modernize .NET Applications with AWS Transform

Lam Bùi — Mon, 22 Sep 2025 14:28:09 +0000

I. Concepts

1. What is AWS Transform?

2. Primary Use Cases

Mainframe Modernization

Transform COBOL, JCL, CICS, Db2, VSAM into modern Java applications.
Automatically analyze code, extract business logic, generate technical documents, and refactor.

VMware Migration

Migrate virtual machines from on‑premises VMware to Amazon EC2.
Auto-generate migration plans, configure VPC networking, and convert VMs to EC2.

.NET Modernization

Upgrade from legacy .NET Framework to cross‑platform .NET.
Integrates with Visual Studio; automatically resolves dependencies, builds, and tests.

Migration Assessment

Pre-migration cost and feasibility assessment.
AI-driven recommendations for EC2 sizing, BYOL optimization, and cost analysis.

3. Key Capabilities

Agentic AI with human-in-the-loop (HITL) for complex tasks and decisions.
End‑to‑end automation: discovery → planning → code refactor → deployment.
Flexible collaboration: role-based access (Admin, Approver, Contributor, Reader).
Workspaces to isolate projects, with dedicated jobs and connectors.
Worklog & dashboards to track progress and decisions.

4. Typical Workflow

1) Environment setup

Create an AWS account with administrative permissions.
Enable the AWS Transform service.
Configure AWS IAM Identity Center for users and groups.

2) Create a Workspace

Dedicated per transformation project.
Contains connectors, jobs, and artifacts.

3) Create a Connector

Connect to target environments (VMware account, GitHub repository, other AWS accounts).
Configure IAM policies and encryption with KMS.

4) Create a Job

Choose job type: VMware Migration, Mainframe Modernization, .NET Transform, etc.
Upload inputs (RVTools export, source code, JCL, spreadsheets, Git repo, etc.).

5) Execute Workflow steps

VMware: Discovery → Grouping → Generate VPC → Replicate & Migrate
Mainframe: Analyze COBOL → Generate Docs → Extract Logic → Refactor → Build Java
.NET: Connect Git → Resolve Dependencies → Transform Code → Test & Build
Assessment: Upload file → Cost estimation → Migration feasibility

6) Track Progress

Worklog: detailed action log.
Dashboard: overall job status and KPIs.

5. Security & Governance

Data encrypted by default using AWS KMS.
Option to use customer‑managed KMS keys for tighter control.
Comprehensive IAM policies for access control, connector creation, and deployments.
Supports cross‑region processing and compliance validation.

6. Roles

Role	Core Permissions
Admin	Full control: create workspace, job, connector; approve HITL steps
Approver	Near‑admin; cannot modify users, connectors, or workspaces
Contributor	Execute jobs, upload files, respond to HITL; cannot approve critical steps
Reader	Read‑only access to progress and artifacts

7. Highlights by Use Case

VMware Migration

AI‑generated AWS VPC topology mirroring on‑prem networks.
Automatic wave planning from discovery data.
Import from NSX or RVTools via collector or direct file upload.
Supports Hub‑and‑Spoke and Isolated VPC topologies.

Mainframe Modernization

Full support for COBOL, JCL, Db2, and VSAM.
Clear stages: analyze → extract logic → refactor to Java.
Re‑run per stage for iterative validation.
Outputs modern Java applications deployable on AWS.

.NET Modernization

Connect GitHub repository to retrieve code.
Automatically resolve missing libraries and dependencies; refactor to .NET 6/7/8.
Integrate with Visual Studio for review and edits.
Build and deploy to test environments.

8. Core Benefits

Accelerate migration/modernization by 50–80%.
Reduce manual errors on complex workloads.
Preserve critical business logic via intelligent extraction.
Scale for large organizations: multi‑account, multi‑region, IAM governance.

II. Step‑by‑Step Guides

1) Modernize .NET Applications with AWS Transform

1.1 Objectives

Transform legacy .NET Framework applications (often Windows‑based) into cross‑platform .NET (Linux‑ready .NET 6/7/8) for deployment on AWS services such as ECS, Lambda, and EC2.

1.2 Prerequisites

Required Item	Description
✅ Source repo for .NET code	GitHub, CodeCommit, GitLab, etc.
✅ Project files identified	Clear structure with .csproj and/or .sln
✅ Dependencies resolved	Can be handled during transform if needed
✅ AWS account with Transform enabled	Use IAM Identity Center for user access

1.3 Procedure

Step 1: Create a Workspace

Open the AWS Transform portal (service URL).
Sign in with IAM Identity Center credentials.
Open your existing workspace or create a new one.

Step 2: Create a Job

In the workspace, select Create a job.

Choose job type: .NET Modernization.

Name the job, optionally add a description, and select Create job.

Step 3: Create a Connector (GitHub)

Choose the AWS account ID used to connect to the source repository and select Next.

Fork the sample .NET project to your own GitHub account: https://github.com/aws-samples/legacy-cycle-store-mvc-app

Navigate to Developer Tools → Connections and select Create connection.

In Create a connection, choose GitHub and provide a Connection Name. Select Connect to GitHub.

In Connect to GitHub, choose Install a new app.

Select Only select repositories, choose the forked .NET repository, then Install & Authorize.

Back in Connect to GitHub, select Connect.

Copy the ARN of the new connection.

Return to AWS Transform and paste the Connection ARN and Connection Name. Select Initiate connector creation.

Choose Copy connector link, open it in a new tab, and approve.

Select Finalize connector to continue.

Step 4: Discover Resources for Transformation

AWS Transform scans the repository to prepare for modernization.

Step 5: Prepare and Start Transformation

In Prepare resources for transformation, review resources and select Confirm repository.

Review settings (branch to create, .NET version, etc.) and select Approve and start transformation.

When finished, verify the generated branch in your GitHub repository.

Forem: Lam Bùi

Building an Intelligent Document Processing Pipeline with AWS Lambda and Bedrock Data Automation

Introduction

What You Will Learn

Solution Architecture

Knowledge Requirements

Article Structure

Prerequisites

Lambda Function Code Reference

Introduction to Amazon Bedrock Data Automation

Intelligent Document Processing with Generative AI

Why Choose BDA?

1. Complete Automation

2. BDA vs Direct Bedrock API

Business Value and Use Cases

Government & Public Sector

Healthcare

Finance & Banking

Logistics & Supply Chain

Retail & E-commerce

Why Combine BDA with Lambda?

Statistics and Impact

AWS Services in This Guide

Amazon Bedrock Data Automation

AWS Lambda

Amazon S3 (Simple Storage Service)

Amazon CloudWatch

AWS IAM (Identity and Access Management)

Detailed Workflow

Difference from Traditional IDP

Part 1: Environment Setup

Step 1.1: Enable Model Access in Amazon Bedrock

Models to Enable:

Implementation Steps:

Step 1.2: Create S3 Buckets

Create Input Bucket:

Create Output Bucket:

Step 1.3: Create IAM Role for Lambda

Select Trusted Entity:

Assign Permissions:

Name and Create Role:

Step 1.4: Create BDA Project

Configure Project:

Part 2: Create Lambda Function

Step 2.1: Create Lambda Function

Step 2.2: Configure Lambda Function

Increase Timeout and Memory:

Step 2.3: Write Lambda Code

Part 3: Configure S3 Trigger

Step 3.1: Add S3 Trigger for Lambda

Part 4: Testing and Deployment

Step 4.1: Prepare Test Document

Step 4.2: Upload and Test

Step 4.3: Check Lambda Execution

Step 4.4: Check Output

Real-World Example: Government Birth Certificate Processing

Use Case from AWS Blog

The Problem

Solution with BDA

Results Achieved

Fields Extracted Automatically

Extending the Solution

Cleanup Resources (Optional)

Delete Lambda Function:

Delete S3 Buckets:

Delete IAM Role:

Delete BDA Project:

Reference Documentation

AWS Documentation:

API References:

AWS Blogs and Case Studies:

BDA-Specific:

IDP with Generative AI:

Advanced Topics:

Solution Guidance:

Product Pages:

Video Resources:

Conclusion

Summary of What You Learned

Next Steps