Forem: Pratik Ponde

Amazon Bedrock Beginner Guide: A Complete Deep Dive into AWS Generative AI

Pratik Ponde — Mon, 30 Mar 2026 07:06:55 +0000

In this guide, we’ll take a beginner-friendly deep dive into Amazon Bedrock, explaining how it works, its features, costs, and use cases.

🔍 How a Simple Problem Introduced Me to Amazon Bedrock

A while ago, I was trying to find a solution for simple problem:

How to make systems understand files and data intelligently, not just match keywords.

When I started exploring AI solutions, things quickly became complicated.
Hosting models, managing infrastructure, and scaling felt like too much work for what I wanted to create.

That’s when I discovered Amazon Bedrock.

What stood out wasn’t just the ability to generate text, it was the flexibility.
It could handle chat, text, images, and more, all through simple APIs. I didn’t have to worry about infrastructure, thanks to Amazon Web Services.

That shift from managing systems to building solutions is what this article is all about.

🧠 What is Amazon Bedrock?

Amazon Bedrock is a fully managed AWS service that allows developers to build and scale generative AI applications using foundation models (FMs).

What are Foundation Models?

Foundation Models are large AI models trained on extensive datasets that can:

Understand text
Analyze images
Generate content
Answer questions

Why Use Bedrock?

No need to manage infrastructure
Built for security and scalability
Cost-effective with a pay-as-you-go model
Seamlessly integrates with the AWS ecosystem

Popular Models Available

Anthropic Claude
Meta Llama
Amazon Titan

🧠 What Can You Do with Amazon Bedrock?

The range of AI capabilities that Amazon Bedrock provides through a single API layer is one of its main advantages.

Let's divide it up into the main categories of capabilities:

💬 1. Chat (Conversational AI)

Chat is the most common way to interact with Amazon Bedrock.
You simply type a question or instruction, and the model responds just like a conversation.
It’s perfect for building chatbots, assistants, or automating responses.

Bedrock supports building intelligent chat systems using models like:

• Claude 3.5 Haiku
• Claude 3 Sonnet

- What you can build:

• Customer support bots
• Internal enterprise assistants
• DevOps copilots

- Example Prompt:

{
  "messages": [
    {
      "role": "user",
      "content": "Explain AWS Bedrock in simple terms"
    }
  ]
}

👉 Best practice:
• Use structured conversation format (messages) instead of plain prompt
• Maintain chat history for context

✍️ 2. Text Generation

Text generation is used when you want the AI to create content.
This can include summaries, articles, code, or structured outputs like JSON.
It’s ideal for automation tasks, documentation, and content creation.

- Supported Models:

• Amazon Titan Text
• Claude 4 Sonnet

- What you can build:

• Blog/article generation
• Code generation
• Email drafting
• Summarization

- Example Use Case:

{
  "prompt": "Write a professional email for leave request",
  "max_tokens": 150,
  "temperature": 0.7
}

🖼️ 3. Image Generation

With image input, you can send pictures to the model and ask questions about them.
The AI can analyze visuals, detect objects, or extract information from images.
This is useful for use cases like document verification, image matching, and visual inspection.

Bedrock also supports text-to-image generation.

- Supported Model:

• Amazon Titan Image Generator

- What you can build:

• AI-generated designs
• Marketing creatives
• Thumbnails and banners

- Example Prompt:

{
  "textToImageParams": {
    "text": "A futuristic city with flying cars at sunset"
  }
}

⚙️ Understanding Bedrock API Parameters

When you work with Amazon Bedrock, you basically send a request to the AI saying:

“Here’s what I want and here’s how I want you to respond.”

A typical request looks like this:

{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 1000,
  "temperature": 0,
  "top_p": 1,
  "messages": [...]
}

Let’s break this down 👇

1. anthropic_version - [ Which rulebook to follow ]

• Defines the API schema version
• It’s required for Anthropic models (like Claude)
• Always keep this fixed unless AWS updates it

2. max_tokens - [ How long should the answer be? ]

This controls the length of the response.

Higher value → longer answer
Lower value → short and precise

Quick tips:

Use 200–500 → for JSON / short answers
Use 800+ → when you want detailed explanations

3. temperature - [ How creative should the AI be? ]

This controls how predictable or creative the response is.

0 → very strict, same answer every time
0.5 → balanced
1 → more creative and random

4. top_p - [ How many options should the AI consider? ]

This one sounds complex, but it’s actually simple.

• Controls token probability sampling
• Limits how “wide” the model thinks
Value Behavior

1 Consider all possibilities
<1 More focused output

Best practice:
• Use top_p = 1 with temperature = 0

5. messages - [ What do you actually want? ]

This is the most important part.

This is where you:

Ask your question
Give instructions
Provide input (text, image, etc.)

"messages": [
  {
    "role": "user",
    "content": [...]
  }
]

📌 Types of input you can send

Inside messages, you can include:

text → normal instructions
image → for image understanding
document → structured files (limited use)

💰 Amazon Bedrock Pricing

When you first look at Bedrock pricing, it can feel confusing but the idea is actually very simple:

You only pay for what you use (pay-as-you-go)

No upfront cost, no subscription required.

1. The Most Important Concept: Tokens

Bedrock pricing is mainly based on tokens.

What is a token?

A token is a piece of text (word or part of a word)
Example: “Hello world” ≈ 2–3 tokens

You are charged for:

Input tokens (what you send)
Output tokens (what AI generates)

2. Pricing Depends on the Model

Different models = different pricing

For example:

Anthropic Claude

~$6 per 1M input tokens
~$30 per 1M output tokens

Smaller / cheaper models

Can be 10x cheaper
Example: some models start around $0.07 per 1M tokens

3. Image Pricing

For image-based tasks:

You are charged per image processed
Example: ~$0.03 to $0.60 per image depending on operation

For detailed pricing information, click here 👉 AWS Bedrock Pricing

✨Final Thoughts

Amazon Bedrock makes it extremely simple to get started with generative AI without worrying about the infrastructure or complexity.
With multiple models available, flexible pricing options, and support for text, chat, and image-based applications, it provides the ability to create highly scalable applications.
The key here is to understand how to effectively tune parameters, select the most appropriate model, and manage costs effectively for your application.

Start simple, experiment, and scale as needed and that’s where Bedrock shines.

🔜 What’s Next

In our next article, we’ll work on a hands-on example of a real-world use case involving Amazon Bedrock and AWS Lambda.
The example will be a solution to verify if a reference image exists within a target file, such as a document or PDF.

💬 Let’s Keep the Conversation Going

Have thoughts, questions, or any experience with Generative AI to share? I would love to hear from you! Feel free to leave a comment or connect with me on LinkedIn. Let's learn and grow together as a community of builders.
Keep exploring, keep automating and see you in the next one!

How to Get Real-Time Notifications When AWS Glue Schema Registry Changes

Pratik Ponde — Tue, 17 Feb 2026 11:45:58 +0000

A real-world DevOps journey with AWS Glue, EventBridge, and Lambda

In this article, I walk through a real-world approach to detecting AWS Glue Schema Registry updates in real time and why making schema changes observable matters in production systems.

🚨The Problem That Started It All

While working as a DevOps engineer, I supported a Kafka-based event-driven system where schemas mattered a lot. Producers and consumers depended heavily on schema versions. Any change, whether intentional or accidental, could quietly disrupt downstream services.

We were already using AWS Glue Schema Registry to manage schemas for Amazon MSK. It provided us with versioning and compatibility checks.

But one question kept coming up during reviews:

“How do we know when someone updates a schema ?”

There was no alert, no trigger, no automation just a silent update sitting in Glue.

That’s when I decided to build an event-driven notification system for schema changes.

💡The Idea: Make Schema Changes Event Driven

Instead of polling Glue or depending on manual communication, the idea was straightforward:
Whenever a schema changes, trigger an action automatically.

And that action Calls a POST API which could:

Notify teams
Trigger validations
Update dashboards
Or kick off CI/CD pipelines

⚙️Comprehensive Architecture Flow

Here’s how the flow looks in real life:

A schema is created or updated in Glue Schema Registry.
CloudTrail records the API activity.
EventBridge listens for specific Glue schema events.
Lambda is triggered automatically.
Lambda calls a POST API with schema details.

No polling. No cron jobs. Fully event-driven.

💻 Source Code and GitHub Repository

The complete implementation for this architecture is available on GitHub.

🔗 GitHub Repository:

pratiksponde / AWS-Glue-Schema-EventBridge-Lambda

How to Get Real-Time Notifications When AWS Glue Schema Registry Changes

Real-time notifications for AWS Glue Schema Registry changes using EventBridge and Lambda.

This repository provides a fully event-driven solution to detect and respond to schema changes in AWS Glue Schema Registry in real time. It uses AWS CloudTrail, Amazon EventBridge, and AWS Lambda to automatically capture schema-related API events and trigger downstream notifications or integrations.

📖 Full article explaining the architecture and implementation:
https://dev.to/pratik_26/how-to-get-real-time-notifications-when-aws-glue-schema-registry-changes-4nbd

🚀 Architecture Overview

This solution follows a serverless, event-driven architecture:

AWS Glue Schema Registry
- Stores and manages schemas for streaming and data applications.
- Events occur when schemas are created, updated, or new versions are registered.
AWS CloudTrail
- Records all Glue Schema Registry API calls such as
  - CreateSchema
  - RegisterSchemaVersion
  - UpdateSchema
- These events are automatically available to EventBridge.
Amazon EventBridge
- Captures relevant Glue events from CloudTrail.
- Filters only schema-related changes.
- Triggers the Lambda function.
AWS Lambda
- Receives…

View on GitHub

Feel free to clone the repository and try it in your own AWS environment.

Now, let’s dive deep into the hands-on implementation step by step.

Step 1: Capturing Schema Changes with CloudTrail

Steps to Create a Default Trail (Console):

Open CloudTrail: Go to the AWS CloudTrail console.
Create Trail: Click Create trail.

3. Configure Name: Enter a trail name (e.g., DefaultTrail).
4. Storage Location: Select Create new S3 bucket to let CloudTrail handle permissions automatically.
5. Log File Validation: Leave enabled (default) to ensure log integrity.

6. KMS Encryption: Leave enabled (default) for security.
7. Finish: Click Next, then Create trail.

AWS Glue APIs like CreateSchema, RegisterSchemaVersion, UpdateSchema are automatically logged in CloudTrail.

This was a big win no custom instrumentation required.

Every schema change already produced a reliable audit event.

Step 2: Filtering the Right Events Using EventBridge

Next, I created an EventBridge rule that listens only to Glue schema related CloudTrail events.

Instead of triggering Lambda for every Glue operation, the rule filters on:

• Event source: glue.amazonaws.com
• Event names related to schema updates

This kept the system clean, efficient, and cost-effective.

Steps to Create a EventBridge Rule (Console):

Open EventBridge: Navigate to the Amazon EventBridge console.
Create Rule: In the navigation pane, choose Rules, and then choose Create rule.

3. Define Rule: Enter a Name and optional Description for the rule and select default Event Bus.
4. Configure the Event Source: Use the visual builder or JSON editor to define your event pattern.

Use below event pattern for AWS Glue schema registry.

  {
    "source": ["aws.glue"],
    "detail-type": ["AWS API Call via CloudTrail"],
    "detail": {
      "eventSource": ["glue.amazonaws.com"],
      "eventName": ["RegisterSchemaVersion", "UpdateSchema", "CreateSchema"]
    }
  }

5. Select Targets: From the Select a target list, choose the AWS Lambda service to invoke when the event is matched & configure the required details for the selected target.

6. Finish: Choose Create rule to activate your rule.

Step 3: Lambda as the Brain of the System

Lambda became the decision-maker and Its responsibilities were very clear:

• Parse the CloudTrail event
• Identify what changed
• Extract schema details (registry, name, version ARN)
• Prepare a meaningful payload
• Call the external POST API

This approach kept Lambda lightweight and focused.

Steps to Create a Lambda Function (Console):

Open Lambda Function: Navigate to AWS Lambda function console.
Create Function: Select Author from scratch, give function name and you can select any runtime (I have selected python 3.11), select create a new role with basic Lambda permissions and click on create.
Attach Permission: Select created lambda and go to configuration and then select permission and click on the role attached to function. Add AWSGlueSchemaRegistryFullAccess permission policy to this role.

Note: Modify the policy as needed to align with your security requirements and best practices.
4. Set Environment Variable: Go to configuration and then select permission and click on environment variables.

Key: API_URL
Value: http://ip.of.instance:port/contextpath/of/your/api
(This is just a sample syntax. Replace this with your actual API Url.)

5. Upload Code: Please upload below given code to call an external API.

import json
import os
import urllib3

http = urllib3.PoolManager()

API_URL = os.environ.get("API_URL")

def extract_schema_info(detail):
    request_params = detail.get("requestParameters", {})
    response_elements = detail.get("responseElements", {})

    schema_arn = response_elements.get("schemaArn")
    schema_version_arn = response_elements.get("schemaVersionArn")

    registry_name = request_params.get("registryId", {}).get("registryName")
    schema_name = request_params.get("schemaName")

    return {
        "registry_name": registry_name,
        "schema_name": schema_name,
        "schema_arn": schema_arn,
        "schema_version_arn": schema_version_arn
    }

def lambda_handler(event, context):
    try:
        detail = event.get("detail", {})
        event_name = detail.get("eventName")


        if event_name not in [
            "CreateSchema",
            "RegisterSchemaVersion",
            "UpdateSchema"
        ]:
            print(f"Ignored event: {event_name}")
            return {"status": "ignored"}

        schema_info = extract_schema_info(detail)

        payload = {
            "eventType": event_name,
            "schemaUpdated": True,
            "schemaDetails": schema_info,
            "timestamp": detail.get("eventTime"),
            "awsAccount": detail.get("recipientAccountId"),
            "region": detail.get("awsRegion")
        }

        encoded_body = json.dumps(payload).encode("utf-8")

        response = http.request(
            "POST",
            API_URL,
            body=encoded_body,
            headers={
                "Content-Type": "application/json"
            },
            timeout=urllib3.Timeout(connect=5.0, read=10.0)
        )

        print("API response status:", response.status)
        print("API response body:", response.data.decode())

        return {
            "status": "success",
            "httpStatus": response.status
        }

    except Exception as e:
        print("Lambda execution failed:", str(e))
        raise

Step 4: Calling a API

The final step was integration and the Lambda sends a POST request to an API with details like:
  • Schema name
  • Registry name
  • Schema version ARN
  • Event type (create/update)
  • Timestamp and region
From here, the API can:
  • Notify consumers
  • Trigger validations
  • Store audit records
  • Or block deployments if needed.

🚀Why This Approach Worked So Well

What I liked most about this solution:
• ✅ Fully event-driven
• ✅ No polling or scheduled jobs
• ✅ Zero impact on producers or consumers
• ✅ Scales automatically
• ✅ Clear audit trail for schema changes
It also aligned perfectly with serverless best practices.

📚Real Lessons Learned

While implementing this, a few things stood out:
• CloudTrail events are rich but noisy → filtering is critical
• Lambda should not assume all fields exist in every event
• Timeouts and retries matter when calling external APIs
• Logging schema ARNs saved a lot of debugging time later
These small details made the difference between a demo and a production ready solution.

✨Final Thoughts

Schema changes are not just metadata updates; they are potential breaking changes. By turning schema updates into events, this setup changed Glue Schema Registry from a passive store into an active part of the system architecture. If you’re running Kafka on AWS and care about schema governance, it’s a good idea to adopt this pattern.

💬 Let’s Keep the Conversation Going

Have thoughts, questions, or any experience with Event driven architectures to share? I would love to hear from you! Feel free to leave a comment or connect with me on LinkedIn. Let's learn and grow together as a community of builders.
Keep exploring, keep automating and see you in the next one!

AWS MSK Using Terraform: Multi-Environment Deployment Guide

Pratik Ponde — Fri, 30 Jan 2026 10:11:07 +0000

👋 Hey there! This is Pratik, a Senior DevOps Consultant with a strong background in automating and optimizing cloud infrastructure, particularly on AWS. Over the years, I have designed and implemented scalable solutions for enterprises, focusing on infrastructure as code, CI/CD pipelines, cloud security, and resilience. My expertise lies in translating complex cloud requirements into efficient, reliable, and cost-effective architectures.
Through this article, I aim to share practical insights into building and managing AWS MSK Serverless using Terraform, helping fellow engineers and teams design scalable, secure, and resilient streaming architectures on AWS.

⚡Amazon MSK: Overview and Types

AWS MSK is a fully managed service that makes it easy to run Apache Kafka on AWS without managing the infrastructure yourself. Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps.

AWS MSK offers two deployment types:

1. Amazon MSK Provisioned

You manage the cluster capacity by choosing instance types and the number of brokers. It offers more control over performance, scaling, and configuration, making it suitable for predictable workloads and production environments that need fine-tuning.

2. Amazon MSK Serverless

AWS automatically manages capacity, scaling, and broker infrastructure. You don’t need to choose instance types or manage brokers. It is ideal for variable or unpredictable workloads and for teams that want minimal operational overhead.

⚙️Core Components and Their Functionality

Broker Nodes: When you create an Amazon MSK cluster, you define the number of broker nodes per Availability Zone (minimum one per AZ). In MSK Provisioned, you can choose between Standard and Express broker types. In MSK Serverless, broker management is handled automatically, and you only configure cluster-level capacity.
ZooKeeper Nodes: Amazon MSK automatically provisions Apache ZooKeeper nodes to support reliable cluster coordination.
KRaft Controllers: KRaft is Kafka’s modern metadata management mode that replaces ZooKeeper. Metadata is managed internally by Kafka controllers, with no additional setup or cost required.
Producers, Consumers, and Topics: You can use standard Kafka operations to create topics and publish or consume data.
Cluster Operations: You manage clusters using the AWS Console, AWS CLI, or SDKs to perform actions such as creating, updating, viewing, or deleting clusters.

🎯Learning Objectives and Hands-On Walkthrough

This article demonstrates how to design and implement reusable Terraform modules for core infrastructure components such as the VPC and Amazon MSK, while supporting multiple environments (for example, Dev and UAT) through dedicated environment configurations using terraform.tfvars and variable definition files. It also covers configuring a remote Terraform backend using Amazon S3 to securely store and manage the Terraform state.
The solution provisions a complete Amazon MSK infrastructure, including the VPC, subnets, security groups, IAM roles, the MSK cluster, and a client EC2 instance. With a small set of Terraform commands, you can reliably create, update, and decommission the entire environment in a consistent and repeatable manner across environments.

Resources Covered in This Guide:

A VPC with public and private subnets across three Availability Zones.
Full networking setup, including Internet Gateway and route tables.
A secure Amazon MSK Serverless cluster.
An EC2 instance configured with Kafka tools and authentication.
IAM roles and security groups to enable secure communication between the EC2 instance and the MSK cluster.

🚀Let’s begin!

📋1. Prerequisites

AWS Account: Ensure you have an AWS account with programmatic access and that your AWS credentials are configured locally using the AWS CLI.

aws configure

S3 Bucket: S3 Bucket for remote backend.
Terraform Setup: Make sure Terraform is installed on your local environment.

Installing Terraform

Terraform is straightforward to set up. The following sections provide installation instructions for the most common operating systems.

On macOS (using Homebrew):

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

On Windows (using Chocolatey):

choco install terraform

On Linux (Debian/Ubuntu):

wget -O- [https://apt.releases.hashicorp.com/gpg](https://apt.releases.hashicorp.com/gpg) | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] [https://apt.releases.hashicorp.com](https://apt.releases.hashicorp.com) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

After installation, verify it's working by running:

terraform --version

💻2. Deep Dive into the Terraform Code

The following directory structure shows the Terraform modules for provisioning the VPC and Amazon MSK Serverless.

Core Networking Resources (VPC, Subnets, Gateways & Route Tables)

This section establishes the core network infrastructure for our environment. We begin by creating a new VPC, followed by provisioning three public and three private subnets one in each AWS Availability Zone to ensure high availability. The Internet Gateway, NAT Gateway, and route tables are configured to provide secure internet connectivity for the EC2 instance.

Security Controls (Security Groups and SSH Access)
- SSH Key: Terraform automatically generates an RSA key pair. The public key is uploaded to AWS using aws_key_pair, while the private key is stored locally as msk-client-key.pem, allowing secure SSH access to the EC2 instance.
- Security Groups: Two security groups are created one for the MSK cluster and one for the EC2 client. The rules are configured to allow unrestricted communication between the EC2 instance and the MSK cluster, while limiting inbound internet access to the EC2 instance to SSH traffic only.

IAM Permissions for EC2

Instead of embedding AWS access keys, we use an IAM role to grant permissions securely. This configuration creates an aws_iam_role that the EC2 instance can assume. An attached aws_iam_policy provides the required permissions to connect to the MSK cluster, describe resources, and read from or write to Kafka topics. This approach follows AWS security best practices and is the recommended way to manage service access.

Amazon MSK Serverless Cluster

This resource creates an Amazon MSK Serverless cluster with a configurable name. It deploys the cluster within the specified subnets and associates it with the MSK security group for secure network access. IAM-based SASL authentication is enabled to allow secure client access using IAM roles. Resource tags are also applied for easier management and identification.

Kafka Client EC2 Instance

This instance acts as the Kafka client for our setup. We provision a t2.micro EC2 instance and use a user_data script that runs automatically during the first boot to configure the environment.
The script performs the following tasks:

Installs Java
Downloads and extracts the required Kafka version
Installs the AWS MSK IAM Authentication library for secure access
Creates the client.properties file with the necessary configuration for IAM-based authentication.

This ensures the EC2 instance is fully prepared to connect to and interact with the MSK cluster.

☁️3. Provisioning the Infrastructure

Note: The full Terraform scripts are available here⬇️:

pratiksponde / AWS-MSK-Terraform-code

This repo contains terraform module code to provision AWS VPC and MSK Serverless cluster

🚀 AWS MSK Using Terraform: Multi-Environment Deployment Guide

This repository contains Terraform infrastructure code to deploy Amazon MSK Serverless across multiple environments such as Dev and UAT using a modular and scalable architecture.

It provisions networking, security, IAM roles, MSK Serverless cluster, and a Kafka client EC2 instance to enable secure Kafka communication.

📖 Full article:
https://dev.to/pratik_26/deploying-amazon-msk-serverless-across-multiple-environments-with-terraform-2i8c

🏗️ Architecture Overview

The infrastructure created by this project includes:

VPC with public and private subnets
Internet Gateway and routing
Amazon MSK Serverless cluster
EC2 instance as Kafka client
IAM roles and policies
Security groups
Remote Terraform state stored in S3

📂 Repository Structure

AWS-MSK-Terraform-code/
├── modules/
│   ├── vpc/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── outputs.tf
│   │
│   └── msk/
│       ├── main.tf
│       ├── variables.tf
│       └── outputs.tf
│
├── environments/
│   ├── dev/
│   │   ├── backend.tf
│   │   ├── main.tf
│   │   ├──

…

View on GitHub

Step 1: Configure the Remote Backend (Dev Environment)

The first step is to configure the remote backend for the Dev environment. This is done by updating the backend configuration under the path:
Environment → Dev → backend.tf
The following configuration uses Amazon S3 to store the Terraform state file securely and enables state locking to prevent concurrent modifications:

terraform {
  backend "s3" {
    bucket        = "your-bucket-name"
    key           = "msk/dev/terraform.tfstate"
    region        = "region-of-bucket"
    use_lockfile  = true
  }
}

This setup ensures that the Terraform state is centrally managed, secure, and consistent when working across environments or teams.

Step 2: Run Terraform from the Correct Environment Directory

Before executing any Terraform commands, ensure that you are in the correct environment directory. For the Dev environment, navigate to:
Environment → Dev

Initialize Terraform

Run the following command to initialize the working directory. This step downloads the required AWS provider plugins and configures the backend.

terraform init

Plan the Deployment

This command performs a dry run and shows a detailed preview of the resources Terraform will create, modify, or delete without making any changes.

terraform plan

Apply the Configuration

This command executes the planned changes and provisions the resources in your AWS account. Confirm the operation by typing yes when prompted.

terraform apply

✅4. Testing and Verifying the Cluster Setup

After terraform apply completes successfully, follow the steps below to validate that the setup is working as expected.

Step 1: Get the EC2 Public IP

Retrieve the public IP address of the EC2 instance from the Terraform outputs.

Step 2: SSH into the EC2 Instance

The private key file msk-client-key.pem is saved in your project directory. Ensure it has the correct permissions:

chmod 400 msk-client-key.pem

Then connect to the instance using SSH:

ssh -i "msk-client-key.pem" ec2-user@<YOUR_EC2_PUBLIC_IP>

Step 3: Get the Bootstrap Brokers String

In your local terminal (not inside the SSH session), retrieve the MSK bootstrap broker’s string:

terraform output bootstrap_brokers

Copy this value. You will use it in the Kafka commands.

Step 4: Create a Kafka Topic

Inside the EC2 SSH session, navigate to the Kafka bin directory and create a topic:

bin/kafka-topics.sh --create \
  --bootstrap-server <bootstrapServerString> \
  --command-config /home/ec2-user/kafka_2.13-3.6.0/bin/client.properties \
  --replication-factor 3 \
  --partitions 1 \
  --topic my-first-topic

Step 5: Start a Producer

In the same terminal, start the Kafka console producer:

bin/kafka-console-producer.sh \
  --broker-list <bootstrapServerString> \
  --producer.config /home/ec2-user/kafka_2.13-3.6.0/bin/client.properties \
  --topic my-first-topic

You will see a > prompt. Type a message such as:

Hello from Terraform!

and press Enter.

Step 6: Start a Consumer (in a New Terminal)

Open another terminal window and SSH into the EC2 instance again. Then run the consumer:

bin/kafka-console-consumer.sh \
  --bootstrap-server <bootstrapServerString> \
  --consumer.config /home/ec2-user/kafka_2.13-3.6.0/bin/client.properties \
  --topic my-first-topic \
  --from-beginning

You should see the message "Hello from Terraform!" appear immediately.
This confirms that your MSK cluster, authentication, and connectivity are working correctly.

🗑️5. Cleaning Up Resources

To avoid unnecessary AWS charges, remember to delete the infrastructure once you are done. One of the advantages of using Terraform is that cleanup can be performed with a single command.

terraform destroy

When prompted, type yes to confirm. Terraform will then safely and systematically remove all resources that were created.

💸Cost Optimization Tips

Use MSK Serverless for variable or unpredictable workloads.
Delete unused topics regularly.
Compress messages to minimize data transfer and storage.
Avoid large message payloads where possible. For detailed pricing information on Amazon MSK, please refer to the link. ⬇️ https://aws.amazon.com/msk/pricing/

💡Final Thoughts

In this article, we designed and implemented a complete Amazon MSK Serverless environment using a modular and reusable Terraform approach. By separating infrastructure into well-structured modules for VPC and MSK, and managing multiple environments such as Dev and UAT with environment-specific configurations and remote state stored in Amazon S3, we achieved a solution that is scalable, maintainable, and aligned with infrastructure-as-code best practices.

This approach not only simplifies provisioning and management but also improves consistency across environments and reduces operational risk. With automated deployment, secure authentication using IAM, and an EC2-based client for validation, you now have a solid foundation for building and operating real-time streaming solutions on AWS.

You can further extend this setup by integrating monitoring, enhancing security controls, and adding CI/CD pipelines to automate infrastructure changes.

✅ Wrapping Up

Thanks for taking the time to explore AWS MSK Serverless with Terraform! I hope this article has helped you understand how to build a scalable, secure, and maintainable streaming data infrastructure. Whether you are a seasoned engineer or just starting with AWS, applying these concepts can make a real difference in managing real-time workloads efficiently.

💬 Let’s Keep the Conversation Going

Have thoughts, questions, or experience with MSK to share? I would love to hear from you! Feel free to leave a comment or connect with me on LinkedIn. Let's learn and grow together as a community of builders.

Keep exploring, keep automating and see you in the next one!

AWS Backup Explained Simply : Use Cases, Setup & Recovery

Pratik Ponde — Mon, 04 Aug 2025 14:42:06 +0000

Through this article, I aim to share practical insights into AWS Backup helping fellow engineers and teams strengthen their cloud resilience.

What is AWS Backup?

AWS Backup is a fully managed backup service that allows you to automate and centrally manage backups across AWS services like EC2, EBS, RDS, S3, DynamoDB, Aurora, and more.
In this guide, we will walk through real use cases and step-by-step instructions to implement and restore backups effectively.

AWS Backup Features Overview

Supports Key AWS Services: EC2, EBS, S3, EFS, RDS, DynamoDB, Aurora, Storage Gateway, and more.
Backup Vault: Central encrypted vault for storing backup copies.
Incremental Backups: Initial full backup followed by incremental changes to reduce storage costs.
Cross-Region Backups: Protect against regional outages.
Automated Backup Plans: Schedule backups on a daily, weekly, or monthly basis.
Cost Estimation: AWS Pricing Calculator helps forecast backup costs.

Use Case 1: Backing Up EC2 Instances

Pre-requisites:

An EC2 Linux instance with a web server (e.g., httpd) installed and a sample index.html file.

Steps:

Create a Backup Plan: Define frequency and retention rules.

Create or Use a Backup Vault: Encrypts data at rest and in transit.

Assign Resources: Choose EC2 instances via tags or specific IDs.

Create IAM Role: Grant necessary permissions (EC2 and AWS Backup full access).

Test Restore:
- Delete the EC2 instance.
- Restore using AWS Backup.
- Confirm the web server (index.html) is running after restore.

Use Case 2: Backing Up S3 Buckets

Pre-requisites:

S3 bucket with versioning enabled.
IAM role with S3 and AWS Backup permissions.

Steps:

Create or configure a versioned S3 bucket.

Set up an on-demand or scheduled backup plan.

Assign S3 resources in the backup plan.
Test Restore:
- Delete objects in the bucket.
- Use the restore job to recover data.
- Monitor the restore job status and verify object restoration.

Note: AWS Backup supports all S3 storage classes except Glacier Deep Archive and some Glacier Flexible Retrieval scenarios.

Use Case 3: Backing Up Databases (RDS/Aurora)

Steps:

Create a new RDS database.
Assign it to a backup plan using tags or direct selection.
Create or attach an IAM role with RDS and AWS Backup permissions.
Perform restore testing by simulating failure or deletion.

Backup Strategy Behind AWS Backup

Full + Incremental: The first backup is full, subsequent backups store only changes.
Efficient Storage: Saves space and time, though full restores may take longer.

Cost Optimization Tips

Use Cost Allocation Tags to track backup related expenses.
Monitor usage and trends using AWS Cost Explorer.
Choose Intelligent Tiering for S3 to balance cost and retrieval needs.
Refer to:
- AWS Backup Pricing

Common FAQs

Q1. Can I back up multiple S3 buckets at once?
Yes, using an automated backup plan. Manual backups are still one bucket at a time.

Q2. How is S3 integrated with AWS Backup?
Backup policies can be applied to S3 with versioning enabled. You can restore to a point in time and define long-term retention policies.

Q3. Is S3 replication a backup method?
No. Replication is for real-time data copying, not version-controlled historical backups.

Final Thoughts

AWS Backup offers a streamlined, secure way to protect your cloud workloads. With automated plans, incremental backups, and centralized management, it simplifies data protection while ensuring compliance and cost control.

✅ Wrapping Up

Thanks for taking the time to explore AWS Backup with me! I hope this article has helped you better understand how to build a reliable and efficient backup strategy for your cloud workloads. Whether you are a seasoned engineer or just getting started with AWS, applying these concepts can make a real difference in your infrastructure's resilience.

🔍 Looking Ahead

This is just one piece of the larger DevOps and cloud automation puzzle. In upcoming posts, I will be diving into topics like infrastructure as code, CI/CD pipelines, and advanced AWS architecture patterns. Be sure to follow for more hands-on guides and real-world insights.

💬 Let’s Keep the Conversation Going

Have thoughts, questions, or your own backup story to share? I would love to hear from you! Feel free to leave a comment or connect with me on LinkedIn. Let's learn and grow together as a community of builders.

Keep exploring, keep automating and see you in the next one!