Forem: Ankit malik

Scaling Amazon Kinesis and AWS Lambda

Ankit malik — Sat, 31 Jan 2026 09:27:30 +0000

Introduction

Amazon Kinesis Data Streams (provisioned mode) and AWS Lambda are commonly paired to build event-driven pipelines. Getting them to scale predictably requires understanding how capacity is allocated (shards), how Lambda consumes shards, and which operational signals reveal problems. This note corrects common terminology and gives concrete actions you can take when streams backlog or show increased latency.

1. Kinesis fundamentals

Shards are the unit of scale.

Per-shard limits (provisioned mode):
- Write: 1 MB/sec or 1,000 records/sec
- Read: 2 MB/sec (shared across consumers)
If input rate approaches shard limits you must increase shard count; if you double shards you roughly double throughput and shard-hour cost.

Partition-key distribution matters.

Low-cardinality keys → hot shards → throttling regardless of total shard count.
Best practice: high-cardinality keys or composite/hash keys to spread load.

Read more about kinesis in my previous article - link

2. Lambda consumption model

Shard-driven parallelism

Each shard is the unit Lambda polls. Parallelism is driven by shard count and the “concurrent batches per shard” setting. This is similar to increasing the number of consumers for other message broker like kafka, rabbitmq etc. We increase consumer in this way.
Rough mapping:
- Max concurrent Lambdas = shard_count × concurrent_batches_per_shard (subject to Lambda account-level concurrency limits).
Ordering: preserved within a batch, not across concurrently processed batches from the same shard.

Batch size and windowing

BatchSize controls how many records Lambda receives per invocation.
MaximumBatchingWindowInSeconds controls how long Kinesis will wait to accumulate a batch.
Trade-offs:
- Larger batch → fewer invocations, higher per-invocation work, higher end-to-end latency.
- Smaller batch → more invocations, lower latency, higher overhead/cost.

3. The parameter: “Concurrent batches per shard”

AWS console & Lambda event source mapping calls it “Concurrent batches per shard” (sometimes displayed as a UI field), not ParallelizationFactor in that terminology.
Behavior:
- Default is 1 (one batch processed at a time per shard).
- Increasing this lets Lambda process multiple batches concurrently from the same shard.
Use-case:
- When a single batch takes long (CPU-heavy work, slow downstream calls), increasing concurrent batches lets you drain the backlog faster without increasing shard count immediately.
Caution:
- Ordering guarantees weaken: ordering is guaranteed only within a batch. If strict ordering across all records is required, increasing concurrent batches may violate your ordering assumptions.

4. How to recover a stuck/lagging stream

If you see many messages backlogged and GetRecords.IteratorAgeMilliseconds rising, you can:

Confirm whether the consumer is the bottleneck (high Lambda duration, throttles, or low concurrency).
If consumer CPU or downstream latency is the cause, increase Concurrent batches per shard to add concurrency per shard and clear backlog faster.
- This increases concurrent Lambda invocations and potentially cost — check account concurrency limits and downstream capacity.
If shard-level throughput is saturated (Write/Read throttles, high WriteProvisionedThroughputExceeded), add shards — do not rely on increasing concurrent batches alone.
If multiple consumers exist and read contention is present, evaluate Enhanced Fan-Out (EFO) to allocate dedicated read throughput per consumer.

Note: Increasing concurrent batches is a fast operational lever to reduce iterator age when consumer-side parallelism is the limiting factor. If Kinesis itself is slow to deliver records, increasing concurrency will not fix it (see Section 6).

5. Provisioned streams — right-size shards for input/output

For provisioned mode, ensure shard count can handle both incoming writes and outgoing reads:
- If incoming rate > shard write capacity → producer-side throttles.
- If consumers require more read throughput than shard-level read capacity → consider more shards or EFO.
Strategy:
- Measure incoming rate (IncomingRecords / IncomingBytes).
- Map required consumer parallelism to shards × concurrent batches per shard, while observing downstream limits.
- Resize shards only when shard-level saturation is the root cause.

6. Key metrics to monitor (CloudWatch), and what they mean

Create dashboards and alerts for these metrics. The ones below are essential:

Kinesis-level

IncomingRecords / IncomingBytes — production or input rates.
GetRecords.Records — records returned per GetRecords.
GetRecords.IteratorAgeMilliseconds — consumer lag (older values = backlog).
GetRecords.Latency - Average (Milliseconds) (a.k.a. GetRecords latency - average in console) — how long Kinesis is taking to respond to GetRecords.
- If this metric rises significantly, it indicates Kinesis is slow to deliver records — this can be an AWS-side or stream-level issue. Increasing consumer concurrency will not help if Kinesis itself is the bottleneck.
WriteProvisionedThroughputExceeded / ReadProvisionedThroughputExceeded — throttling on shard capacity.
You can create this type of cloudwatch metrics dashboard to to check if there is any issues similar to this screenshot where read latency is increased which caused fewer reads.

Lambda-level

ConcurrentExecutions — how many Lambdas are active; should correlate with shards × concurrent_batches_per_shard.
IteratorAge (Lambda) — similar to Kinesis iterator age; useful cross-check.
Duration and Throttles — consumer-side slowness and throttling.
Invocation count and error/ DLQ metrics.

7. GetRecords latency — special attention

GetRecords latency - average (Milliseconds) measures how long Kinesis takes to return records for a GetRecords call.
Operational interpretation:
- A rising trend can mean Kinesis is under pressure, experiencing internal delays, or there are transient AWS-side issues. When this increases:
  - Check AWS service health & region notifications.
  - Check whether many consumers are contending for read throughput.
  - Investigate shard-level throttling.
- If GetRecords latency is high and IteratorAgeMilliseconds continues to increase despite raising consumer concurrency, this points to Kinesis/service-side delivery problems rather than your consumer.
- To resolve this quickly you can increase “Concurrent batches per shard” which can help process the records faster.

8. Cost considerations

Increasing shards increases Kinesis shard-hour costs linearly.
Increasing concurrent batches per shard increases Lambda concurrency and invocation count — this increases Lambda costs (GB-seconds × invocations).
EFO brings additional Kinesis costs per consumer.

Conclusion

Use Concurrent batches per shard (not the misnamed ParallelizationFactor) to increase parallelism per shard when consumer-side slowness is the cause of backlog.
Right-size shards for your input and output workloads — don’t rely exclusively on increasing Lambda concurrency.
Monitor GetRecords.IteratorAgeMilliseconds and GetRecords.Latency - Average (Milliseconds) closely: iterator age shows backlog; GetRecords latency indicates whether Kinesis itself is slow to deliver messages.
Build a focused CloudWatch dashboard with the panels and alerts above so you can confidently distinguish consumer-side vs Kinesis-side issues and act accordingly.

What is Amazon Kinesis Data Streams

Ankit malik — Sun, 25 Jan 2026 08:25:47 +0000

Amazon Kinesis Data Streams (KDS) is a managed AWS service for collecting, processing and storing real-time streaming data at scale. If your application needs to ingest continuous streams of events (logs, clicks, telemetries, sensor data, audit events) and process them with low latency, Kinesis Data Streams is one of the standard options on AWS.

Why use a streaming data store?

You have data arriving continuously (millions of small messages per minute).
You need near real-time processing (analytics, alerts, enrichment, ETL).
You want to decouple producers and consumers (producers write events; many different consumers can read and process them).
You want to replay or reprocess historical data when needed.

Kinesis Data Streams gives you a durable, ordered buffer that sits between producers and consumers so you can build streaming pipelines reliably.

Basic concepts

Stream

A stream is the top-level object. Producers put records into a stream. Consumers read records from it.

Shard

A stream is made of one or more shards. A shard is a unit of capacity — it controls how much data you can write to and read from the stream. You scale a stream by adding or removing shards (resharding).

Record

A record is a single unit of data that producers write. It contains:

a data blob (binary or text),
a partition key,
a sequence number (assigned by Kinesis when the record is accepted).

Partition key

Producers supply a partition key with each record. Kinesis uses the partition key to determine which shard will store the record. Records with the same partition key go to the same shard and preserve ordering relative to each other.

Sequence number

Each record in a shard receives a unique sequence number. Consumers use sequence numbers for reading and for checkpoints (to resume processing).

Retention

Kinesis stores data for a configurable retention period. That means consumers can re-read and replay data within that window.

How data flows (high level)

Producers write records to the stream using the AWS SDK or PutRecord/PutRecords API.
Kinesis routes each record to a shard using the partition key.
Within a shard, records are ordered and get sequence numbers.
Consumers read shards using GetRecords (polling), Enhanced Fan-Out (push to registered consumers), or via managed integrations such as AWS Lambda.
Consumers process the records and checkpoint progress (so they can resume from where they left off).

Consumers and consumption models

Standard (shared) consumers

Consumers poll the shard endpoints for records. Multiple consumers share the shard read throughput.

Enhanced Fan-Out (EFO)

With EFO, each registered consumer gets a dedicated HTTP/2 connection and receives records with very low latency and without sharing read throughput. This is useful when several independent applications need the same stream data.

Kinesis Client Library (KCL)

KCL is a client-side library (Java, also wrappers for other languages) that helps with shard leasing, checkpointing, rebalancing and fault tolerance. Use it when you need an auto-managed consumer group behavior similar to Kafka consumers.

AWS Lambda integration

You can configure Lambda as a consumer. Lambda will poll the stream and invoke your function with batches of records. This provides a serverless way to process stream data but you need to manage batch sizes and function concurrency.

Scaling and resharding

To increase capacity, add shards. To decrease, merge shards.
Resharding is done by splitting a shard into two or merging two shards into one.
Partition key design is important: if keys are skewed, traffic concentrates on a small set of shards and causes hot-shard problems. Prefer even hashing and avoid a single hot key.
For sudden spikes, consider provisioning more shards proactively or design producers/consumers to handle transient throttling.

Ordering and replay

Ordering is preserved per partition key (i.e., within a shard). There is no global ordering across all shards.
Because Kinesis retains data for the configured retention period, consumers can reprocess old data by reading from an older sequence number or timestamp. This is useful for replays, bug fixes, and reprocessing pipelines.

Reliability, security, and durability

Data is replicated across multiple availability zones (managed by AWS) to provide durability.
Access is controlled via IAM policies (who can put, get, or manage streams).
You can enable encryption at rest (KMS) and data in transit is TLS-protected.
Use proper IAM roles for producers and consumers; give least privilege.

Cost (overview)

Kinesis Data Streams cost is primarily driven by:

shard hours (how many shards and for how long),
data ingestion (PUT operations / payload),
optionally enhanced fan-out consumers,
data retrieval or extended retention if you keep data longer.

For cost-sensitive workloads, tune shard count, batch records with PutRecords, and consider Firehose or MSK if they are a better fit.

10. Best practices (short list)

Design partition keys to spread load evenly across shards.
Use PutRecords (batch API) instead of many single PutRecord calls.
Monitor consumer lag and scale shards before lag becomes critical.
Use KCL or a managed consumer pattern to handle shard rebalancing and checkpointing.
Keep messages small and carry only necessary data; offload large payloads to S3 if needed.
Use IAM, encryption, VPC endpoints (if needed), and CloudWatch for observability.

11. When to choose Kinesis Data Streams

Choose Kinesis Data Streams when you need:

ordered, durable streaming with replay capability,
multiple independent consumers reading the same stream,
tight control over shard-level capacity and scaling,
integration with AWS tools (Lambda, Kinesis Data Analytics, Firehose).

Conclusion

Amazon Kinesis Data Streams is a robust, AWS-managed building block for real-time streaming pipelines. It gives you ordered, durable streams, multiple consumption options, and the ability to replay data. For technical teams building event-driven or streaming analytics systems, Kinesis Data Streams is a practical and well-integrated choice on AWS — provided you design partition keys, shards and consumers carefully.

Python: How to Encrypt and Decrypt with AES

Ankit malik — Tue, 23 Sep 2025 00:57:52 +0000

Introduction

Sometimes we need to keep data secret—like passwords, personal details, or private messages. AES (Advanced Encryption Standard) is a very popular way to do this.

AES is a method of turning normal text into unreadable text (encryption) and then back to normal (decryption) using the same secret key (symmetric algorithm).

When you need to protect sensitive information—such as passwords, financial data, or confidential messages—encryption is essential.

Installation

Before you start:

Install Python 3.7 or newer.
Install the Python package cryptography by running:
```
pip install cryptography
```

Understanding AES Encryption

AES is a symmetric key algorithm, which means the same secret key is used for both encryption and decryption.

Key sizes supported by AES:

128 bits
192 bits
256 bits (the strongest and what we use in this example)

Modes of operation decide how the algorithm works internally. Common modes include:

ECB (Electronic Codebook): Simple but not very secure.
CBC (Cipher Block Chaining): Better than ECB but needs extra care for integrity.
GCM (Galois/Counter Mode): Modern and secure. It gives both encryption and a built-in check to ensure the data hasn’t been changed.

In our code, we will use GCM mode.

Python Code Example

Here is a complete example:

import random
import string
import base64
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

def encrypt_with_aes(input: str, enc_key: str, iv: str):
    key = enc_key.encode()
    nonce = iv.encode()
    plaintext = input.encode()
    aesgcm = AESGCM(key)
    ciphertext = aesgcm.encrypt(nonce, plaintext, None)

    # Change the encrypted bytes to a readable Base64 string
    ciphertext_str = base64.b64encode(ciphertext).decode()
    return ciphertext_str

def decrypt_with_aes(input: str, enc_key: str, iv: str):
    key = enc_key.encode()
    nonce = iv.encode()
    ciphertext = base64.b64decode(input)
    aesgcm = AESGCM(key)
    decrypted = aesgcm.decrypt(nonce, ciphertext, None)
    return decrypted.decode()

def generate_iv_string(length=16):
    # Create a random string for the nonce
    chars = string.ascii_letters + string.digits + "#$()*+,-.:;<=>?@[]_"
    return ''.join(random.choices(chars, k=length))

enc_key = "1Xt5YfM4ZNuFdwp3OfVkwkhhQLagWKtt"  # 32-character secret key
iv = generate_iv_string(12)  # make a random nonce/iv
input = "This is a top secret message"

ciphertext = encrypt_with_aes(input, enc_key, iv) # base64 encoded data
print("Ciphertext:", ciphertext)

decrypted = decrypt_with_aes(ciphertext, enc_key, iv)
print("Decrypted:", decrypted)

How the code works:

encrypt_with_aes: Takes the message and makes it unreadable.
decrypt_with_aes: Turns the unreadable message back to normal text.
generate_iv_string: Creates a new random nonce each time.

When you run it, you will see:

Ciphertext: I1M8nE7HxHlmv7uKZPM/FsorN4hIiNhAm8fg2TavM75Dxp00zFrgRQem67E=
Decrypted: This is a top secret message

Tips for Security

Keep the key safe: Don’t write the key directly in your real code. Store it in environment variables or a secure vault.
Use a different nonce every time: Never reuse the same nonce with the same key.
Change keys over time: For long-term projects, rotate (change) your keys regularly.

Conclusion

AES-GCM is a strong and trusted way to keep data safe.

With a few lines of Python, you can hide a message and later get it back using the same key.

To learn more, check the cryptography library documentation.

How to use reduce function in Python

Ankit malik — Thu, 18 Sep 2025 20:55:15 +0000

Introduction

In Python, the reduce function is a powerful tool for performing cumulative operations on iterable data, such as lists or tuples. Instead of writing a loop to repeatedly apply a function to elements, reduce lets you “reduce” the iterable into a single value by successively combining elements. This makes your code more concise and functional in style.

Syntax and Parameters

The reduce function is available in the functools module, so you must import it first:

from functools import reduce
reduce(function, iterable[, initializer])

Parameters:

function: A function that takes two arguments and returns a single value.
iterable: The sequence (list, tuple, etc.) to process.
initializer (optional): A starting value. If provided, the reduction starts with this value; otherwise, the first item of the iterable is used.

Return value: A single value that is the cumulative result after applying the function across all elements.

Coding Examples

While reduce is often used with lambda functions but for learning purpose I am not going to user lambda function here.

Example 1: Summing Numbers

from functools import reduce

def add(x, y):
    return x + y

numbers = [2, 4, 6, 8]
result = reduce(add, numbers)
print(result)  # Output: 20

Here, the add function is applied pairwise: ((2 + 4) + 6) + 8.

Example 2: Finding the Product of a List

from functools import reduce

def multiply(x, y):
    return x * y

numbers = [1, 3, 5, 7]
product = reduce(multiply, numbers)
print(product)  # Output: 105

Example 3: Concatenating Strings

from functools import reduce

def join_strings(a, b):
    return a + " " + b

words = ["Python", "reduce", "function", "tutorial"]
sentence = reduce(join_strings, words)
print(sentence)  # Output: "Python reduce function tutorial"

Example 4: Using an Initializer

from functools import reduce

def add(x, y):
    return x + y

numbers = [10, 20, 30]
result = reduce(add, numbers, 100)  # Starts with 100
print(result)  # Output: 160

Real-World Use Cases

Data Processing: Summing sales figures, combining CSV rows, or aggregating sensor readings.
Mathematical Computations: Calculating factorials or cumulative products.
String Manipulation: Joining words or formatting complex strings.
Custom Aggregations: Reducing logs, merging dictionaries, or performing multi-step computations.

Best Practices

Prefer built-in functions like sum() or max() when possible—they are more readable and optimized.
Use named functions instead of lambdas for clarity, especially in complex operations.
Always import from functools, as reduce is not a built-in function in Python 3.
Provide an initializer to avoid errors when the iterable is empty.
Keep functions pure: Avoid side effects to ensure predictable results.

Conclusion

The reduce function in Python simplifies operations that require the combination of iterable elements into a single value. By understanding its syntax and applying it with clearly defined functions, you can write cleaner, more functional-style code. Whether you’re aggregating data, performing mathematical calculations, or manipulating strings, reduce is a valuable tool for elegant and concise Python programming.

How to Use Partial Function in Python

Ankit malik — Tue, 16 Sep 2025 21:16:44 +0000

Introduction

In Python, functions are first-class objects, which means they can be passed around, modified, or even treated as variables. The functools module provides a tool called partial function, which allows you to fix a certain number of arguments of a function and generate a new function with fewer parameters. This is especially useful when working with repetitive function calls where many arguments remain the same.

Use Cases

Partial functions are commonly used in scenarios where:

Pre-filling function parameters: You want to reuse a function with some arguments fixed, avoiding repeated input.
Simplifying callbacks: Passing simplified functions to APIs, GUIs, or event-driven frameworks.
Improving code readability: Defining specialized versions of general-purpose functions, such as formatting or mathematical calculations.
Data processing pipelines: When mapping or filtering with functions that require multiple arguments, a partial function can simplify the structure.

Code Example with Explanation

Let’s explore an example.

from functools import partial

# A normal function that multiplies two numbers
def multiply(x, y):
    return x * y

# Creating a partial function to always multiply with 2
double = partial(multiply, 2)

# Creating another partial function to always multiply with 5
five_times = partial(multiply, 5)

# Using the partial functions
print(double(10))       # Output: 20 (2 * 10)
print(five_times(4))    # Output: 20 (5 * 4)

Explanation:

The function multiply(x, y) takes two arguments.
By applying partial(multiply, 2), we fix x = 2. The new function double(y) only needs one argument.
Similarly, five_times(y) will always multiply numbers by 5.
This approach eliminates rewriting separate functions for these common operations.

Another real-world example with formatting:

from functools import partial

def format_text(text, prefix, suffix):
    return f"{prefix}{text}{suffix}"

# Create specific formatters using partial functions
add_brackets = partial(format_text, prefix="[", suffix="]")
add_quotes = partial(format_text, prefix='"', suffix='"')

print(add_brackets("Python"))   # Output: [Python]
print(add_quotes("Hello"))      # Output: "Hello"

This technique is particularly handy when creating specialized behaviors without redefining functions.

Conclusion

Partial functions in Python, provided by the functools module, are a clean and efficient way to reduce code duplication. By fixing certain arguments of a function, they let you create customized versions of existing functions for specific use cases. Whether for callbacks, formatting, or data processing, partial functions help write cleaner and more reusable code.

Kubernetes Cheatsheet

Ankit malik — Fri, 23 May 2025 07:40:24 +0000

Introduction

I often forget these kubernetes or K8s commands. So I have created this sheet to quickly look any command.
It might be helpful for you to quickly look for any command. You can bookmark this blog for quick search.

📦 Kubectl Basics


kubectl version                             # Show kubectl & cluster version
kubectl config view                         # View kubeconfig details
kubectl get nodes                           # List all cluster nodes
kubectl get all                             # List all resources in the current namespace
kubectl get pods -A                         # List all pods in all namespaces
kubectl get services                        # List all services

🚀 Pods


kubectl run nginx --image=nginx             # Create a new pod
kubectl get pods                            # List all pods
kubectl describe pod <pod-name>             # Detailed info about a pod
kubectl logs <pod-name>                     # View logs
kubectl exec -it <pod-name> -- /bin/bash    # Shell into a pod
kubectl delete pod <pod-name>               # Delete a pod

🏗️ Deployments

kubectl create deployment myapp --image=myimage    # Create a deployment
kubectl get deployments                            # List deployments
kubectl scale deployment myapp --replicas=3        # Scale deployment
kubectl edit deployment myapp                      # Edit deployment config
kubectl rollout status deployment/myapp            # Check rollout status
kubectl rollout undo deployment/myapp              # Rollback deployment
kubectl delete deployment myapp                    # Delete deployment

🌐 Services

kubectl expose deployment myapp --type=ClusterIP --port=80     # Create service
kubectl get svc                                                 # List services
kubectl describe svc myapp                                      # Service details
kubectl delete svc myapp                                        # Delete service

Service Types: ClusterIP (default), NodePort, LoadBalancer, ExternalName

🛠️ ConfigMaps & Secrets


kubectl create configmap myconfig --from-literal=key=value
kubectl get configmaps
kubectl describe configmap myconfig

kubectl create secret generic mysecret --from-literal=password=mypassword
kubectl get secrets
kubectl describe secret mysecret

📂 Namespaces


kubectl get namespaces
kubectl create namespace dev
kubectl config set-context --current --namespace=dev

📄 YAML Manifest Commands


kubectl apply -f config.yaml           # Apply from file
kubectl delete -f config.yaml          # Delete resources from file
kubectl create -f config.yaml          # Create resources from file
kubectl explain pod                    # Explain resource structure

📊 Monitoring & Debugging


kubectl top pod                        # Show resource usage
kubectl describe node <node-name>     # Node info
kubectl get events                    # Cluster event

📚 Bonus: Useful Shortcuts


kubectl get po                         # po = pods
kubectl get deploy                     # deploy = deployments
kubectl get svc                        # svc = services
kubectl get ns                         # ns = namespaces

API Gateway vs Service Mesh

Ankit malik — Thu, 22 May 2025 15:56:04 +0000

Introduction

As organizations adopt microservices architectures to improve scalability, agility, and team autonomy, managing service communication becomes a central concern. Two popular infrastructure components that help tackle these challenges are API Gateways and Service Meshes. While both deal with communication and traffic management in distributed systems, they serve distinct roles and operate at different layers of the application stack.

In this article, we’ll explore the key differences between API Gateways and Service Meshes, their purposes, features, and when you might use one—or both—in your architecture.

🚪 What is an API Gateway?

An API Gateway is the single entry point for external clients (such as mobile apps, frontend applications, or third-party systems) to access backend services in a microservices architecture. It acts as a reverse proxy that accepts API calls, applies policies, and routes requests to the appropriate services.

🔑 Core Responsibilities:

Routing and forwarding external requests
Authentication and authorization (e.g., OAuth, JWT)
Rate limiting and throttling
Request transformation (e.g., path rewriting, protocol translation)
Aggregating responses from multiple services
Caching and load shedding
Logging and monitoring API usage

🧰 Common API Gateway Solutions:

AWS API Gateway
Kong
Apigee
NGINX
Traefik

📌 Example Use Case:

A mobile app sends a request to /user/profile. The API Gateway:

Authenticates the request using a JWT token.
Applies rate limiting rules.
Routes the request to the User Service.
Aggregates additional data from a Preferences Service (if needed).
Returns the response to the client.

🔄 What is a Service Mesh?

A Service Mesh is an infrastructure layer designed to handle internal service-to-service communication. It decouples networking logic from application code and provides fine-grained control over how services discover and interact with each other.

Service Meshes typically use sidecar proxies (like Envoy) that run alongside each service instance to manage network traffic.

🔑 Core Responsibilities:

Secure service-to-service communication with mutual TLS (mTLS)
Fine-grained traffic routing and load balancing
Automatic retries, timeouts, and circuit breakers
Distributed tracing and telemetry
Policy enforcement and access control
Fault injection and chaos testing

🧰 Popular Service Mesh Solutions:

Istio
Linkerd
Consul Connect
Kuma
AWS App Mesh

📌 Example Use Case:

Service A wants to call Service B. The service mesh:

Intercepts the request via a sidecar proxy.
Applies mTLS for encrypted communication.
Enforces retry logic and timeout settings.
Logs the request metadata for distributed tracing.
Forwards the request to Service B’s sidecar, which passes it to the app.

🆚 API Gateway vs. Service Mesh: Key Differences

Feature	API Gateway	Service Mesh
Primary Role	Manage traffic between clients and services	Manage internal service-to-service traffic
Traffic Direction	North-South (external ↔ internal)	East-West (internal ↔ internal)
Deployment	At the edge of the network	As sidecars alongside services
Authentication	OAuth, API keys, JWT	mTLS, service identity
Traffic Policies	Rate limiting, throttling	Retries, timeouts, circuit breaking
Observability	API usage metrics	Service-level telemetry and tracing
Complexity	Moderate	Higher (due to mesh orchestration)

🧩 Can You Use Both Together?

Yes—and in fact, many production-grade microservice architectures combine both to handle different types of traffic:

The API Gateway sits at the edge of your system, managing communication from external clients.
The Service Mesh governs communication within the system, between internal services.

This layered approach provides full control over all traffic flows, end-to-end security, and improved observability.

✅ When to Use What?

Scenario	Use API Gateway	Use Service Mesh
You need to expose services to the public	✅	❌
You want mTLS between internal services	❌	✅
You want retry logic and circuit breaking	❌	✅
You need OAuth2 authentication for APIs	✅	❌
You want telemetry for service-to-service calls	❌	✅
You need to aggregate multiple services for a client	✅	❌

🧠 Final Thoughts

Both API Gateways and Service Meshes are essential tools in a microservices toolkit—but they serve very different purposes.

Use an API Gateway to secure, route, and manage client access to your services.
Use a Service Mesh to secure, control, and observe internal service-to-service communication.

By understanding their roles, you can build more secure, resilient, and observable distributed systems.

Golang - How to do Benchmarking

Ankit malik — Thu, 15 May 2025 12:26:58 +0000

Performance matters — especially when dealing with large-scale systems or performance-critical code. Go (Golang) provides first-class support for benchmarking via the testing package. This article walks you through the fundamentals of benchmarking in Go: what it is, why it's important, and how to run benchmarks to compare multiple functions.

🧠 What is Benchmarking?

Benchmarking is the process of measuring how efficiently a piece of code executes. In Go, this typically involves determining how long a function takes to run, how many resources it uses, or how well it scales under load.

Benchmarking is different from testing — it doesn’t check if your code is correct but it checks how well it performs.

✅ Why We Should Do Benchmarking

Benchmarking is useful when you want to:

Compare different implementations of the same logic (e.g., using + vs strings.Builder for string concatenation).
Find performance bottlenecks in your application.
Track regressions over time, especially in performance-sensitive systems.
Make data-driven decisions when optimizing code.

In short, benchmarking gives you quantitative insights into your code’s behavior under stress.

💻 Example

Let’s take an easy and real world example which we often overlook

package benchmark

import (
    "strings"
    "testing"
)

func concatWithPlus(a, b string) string {
    return a + b
}

func concatWithBuilder(a, b string) string {
    var sb strings.Builder
    sb.WriteString(a)
    sb.WriteString(b)
    return sb.String()
}

func BenchmarkConcatWithPlus(b *testing.B) {
    for i := 0; i < b.N; i++ {
        _ = concatWithPlus("Hello", "World")
    }
}

func BenchmarkConcatWithBuilder(b *testing.B) {
    for i := 0; i < b.N; i++ {
        _ = concatWithBuilder("Hello", "World")
    }
}

▶️ Running the Benchmark

Save the file as concat_test.go and run:

go test -bench=.

📤 Output

Example output might look like:

goos: linux
goarch: amd64
BenchmarkConcatWithPlus-8        10000000               150 ns/op
BenchmarkConcatWithBuilder-8     20000000                90 ns/op
PASS
ok      example.com/benchmark   2.345s

📖 How to Read the Output

Each line shows:

Field	Meaning
`BenchmarkConcatWithPlus-8`	Benchmark function run on 8 logical CPUs
`10000000`	Number of iterations Go used for benchmarking
`150 ns/op`	Average time taken for one operation (in nanoseconds)

In this case:

concatWithBuilder is faster (90 ns/op) than concatWithPlus (150 ns/op).
Therefore, strings.Builder is a more efficient way to concatenate strings in this context.

Optional Memory Metrics

To view memory allocations and usage:

go test -bench=. -benchmem

Sample output:


BenchmarkConcatWithPlus-8        10000000               150 ns/op            16 B/op          1 allocs/op
BenchmarkConcatWithBuilder-8     20000000                90 ns/op             0 B/op          0 allocs/op

Field	Meaning
`150 ns/op`	How many bytes of memory were allocated per iteration.
`1 allocs/op`	How many memory allocation events occurred per iteration.

This shows:

concatWithPlus allocates memory (16 bytes per operation).
concatWithBuilder avoids allocations — a big win for performance.

🏁 Conclusion

Benchmarking in Go is easy to set up and incredibly valuable. It helps you:

Understand the performance characteristics of your code
Choose the most efficient implementations
Catch regressions before they ship to production

By comparing two simple functions, we saw how small differences in implementation can lead to measurable performance improvements.

What is redis pipeline

Ankit malik — Sun, 11 May 2025 10:53:01 +0000

Introduction

Redis is a fast, in-memory data store widely used for caching, message brokering, and real-time analytics. While Redis is incredibly efficient, network overhead can become a bottleneck when a large number of commands are sent sequentially. This is where Redis pipelining comes in.

Redis pipelining allows a client to send multiple commands to the server without waiting for individual responses. The server processes the batch of commands and returns all responses in a single reply, significantly reducing the latency caused by round-trip communication.

If you are struggling to remember redis commands then this cheatsheet might help you.

Benefits

Redis pipelining offers several advantages:

Reduced Latency: By sending multiple commands in a single request, you cut down on network round-trips.
Improved Throughput: It enables higher command throughput, especially for bulk operations.
Efficient Network Utilization: Reduces packet overhead and increases the efficiency of TCP connections.
Atomicity (optional with MULTI/EXEC): When used inside transactions, pipelining can contribute to atomic command execution.
Avoids Race condition: If we are in some situation like we want to read some counter value and then increment it. Pipeline would help there by reducing the time and run both commands at same time

However, pipelining doesn’t provide isolation like transactions—it simply groups commands for performance.

Code Example - Golang

Perform various type of operations with pipeline -

package main

import (
    "context"
    "fmt"
    "github.com/redis/go-redis/v9"
)

func main() {
    ctx := context.Background()
    rdb := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    })

    pipe := rdb.Pipeline()

    // Queue up multiple commands
    incr1 := pipe.Incr(ctx, "counter1")
    incr2 := pipe.Incr(ctx, "counter2")
    pipe.Set(ctx, "key1", "value1", 0)

    // Execute all commands
    _, err := pipe.Exec(ctx)
    if err != nil {
        panic(err)
    }

    // Access responses
    fmt.Println("counter1:", incr1.Val())
    fmt.Println("counter2:", incr2.Val())
}

Output:

counter1: 1
counter2: 1

Explanation:

rdb.Pipeline() creates a pipeline.
Commands are queued (Incr, Set, etc.).
Exec() sends all commands and receives the results.
Each response is accessed via the queued command result objects.

Code Example - Python

import redis

r = redis.Redis(host='localhost', port=6379, db=0)

# Create a pipeline
pipe = r.pipeline()

# Queue multiple commands
pipe.incr('counter1')
pipe.incr('counter2')
pipe.set('key1', 'value1')

# Execute all commands
responses = pipe.execute()
print("Responses:", responses)

Output:

Responses: [2, 2, True]

Explanation:

pipeline() creates a pipeline object.
Commands are queued with incr and set.
execute() sends all at once and returns results in order.

Conclusion

Redis pipelining is a powerful feature that optimizes the communication between client and server by batching multiple commands. This leads to faster operations and improved efficiency, especially in use cases involving bulk writes or reads. While pipelining does not ensure atomic execution like transactions, it significantly boosts performance for high-volume interactions with Redis.

VsCode Debugger for Ruby on Rails

Ankit malik — Sat, 10 May 2025 10:22:59 +0000

Steps to Add debugger

Install rdbg extention here.
Create launch.json file in vscode.
Add the given configuration to run the code
[Optional] You can try this command also if this is working then it should run rdbg --open -n -c -- bundle exec rails s

`launch.json` file for `vscode`

{
        "version": "0.2.0",
        "configurations": [
                 {
                        "type": "rdbg",
                        "name": "Run rake test",
                        "request": "launch",
                        "command": "rake",
                        "script": "test", 
                        "args": [],
                        "askParameters": false 
                },
                {
                        "type": "rdbg",
                        "name": "Debug Rails server",
                        "request": "launch",
                        "command": "bin/rails",
                        "script": "server", // Launch Rails server with debugger
                        "args": ["-p", "3000"], // Optional: specify port
                        "askParameters": false
                }
        ]
}

Details Reference

https://www.youtube.com/watch?v=e_RKkgiimXE

Redis cheatsheet

Ankit malik — Sat, 10 May 2025 07:40:39 +0000

Introduction

Redis is a powerful in-memory data structure store, commonly used as a database, cache, and message broker.

This is a quick reference guide containing the most commonly used Redis commands. If you are like me who often forget about commands when not using redis actively then it would be helpful.

Cheatsheet Commands

1. String

Use: Store simple values (text, numbers, JSON, etc.)
Commands:

SET key value
GET key
INCR key     # increment number
DECR key     # decrement number
APPEND key value
DEL key

2. List

Use: Ordered list of strings (like a queue or stack)
Commands:

LPUSH key value   # add to head
RPUSH key value   # add to tail
LPOP key          # remove from head
RPOP key          # remove from tail
LRANGE key 0 -1   # get entire list

3. Set

Use: Unordered collection of unique strings
Commands:

SADD key value
SREM key value
SMEMBERS key
SISMEMBER key value
SUNION key1 key2
SINTER key1 key2

4. Sorted Set (ZSet)

Use: Like a set, but each value has a score for sorting
Commands:

ZADD key score member
ZRANGE key 0 -1        # get sorted elements
ZREM key member
ZSCORE key member
ZRANK key member

5. Hash

Use: Store field–value pairs (like a mini JSON or dict)
Commands:

HSET key field value
HGET key field
HGETALL key
HDEL key field
HMSET key field1 val1 field2 val2

6. Fetch All keys

KEYS *           # all keys
KEYS user:*      # keys that start with "user:"
KEYS *:profile   # keys that end with ":profile"
KEYS order:?     # keys like order:1, order:2 (one character wildcard)
KEYS session:[a-z]*  # regex-style pattern (for lowercase a-z)

Scan is production safe

SCAN 0 MATCH user:* COUNT 100

🔹 7. Hyperloglog Commands

Uses hyperloglog data structure under the hood. This helps in cardinality of any key and their members.
For example - You want to count the unique number likes for social media post.

# Add a like - 
PFADD <key> <element>
PFADD like:hll:post_123 user_456

# Get unique like count - 
PFCOUNT <key>
PFCOUNT like:hll:post_123

# Merge HLLs (e.g. across datacenters)
PFMERGE like:hll:post_123:global like:hll:post_123:dc1 like:hll:post_123:dc2

What is Pydantic in python

Ankit malik — Thu, 08 May 2025 11:45:31 +0000

Introduction

In Python, data validation and settings management are common challenges, especially when working with APIs, configuration files, or user inputs. While Python is a dynamically typed language, developers often need ways to ensure that data provided should be of expected types. This is where Pydantic comes into picture.

Pydantic is a data validation and parsing library that uses Python type annotations to validate structured data. Built on top of Python 3.6+ type hints, Pydantic provides automatic data validation and conversion, making it especially popular in FastAPI and other modern Python frameworks.

Why Use Pydantic

Here are some reasons why Pydantic is a go-to tool for developers:

Data Validation: Automatically validate input data using standard Python types.
Error Reporting: Provides clear, structured error messages when validation fails.
Performance: Pydantic is built using dataclasses and is extremely fast.
Type Coercion: Automatically converts compatible types (e.g., string to integer).
Integration: Plays well with modern Python frameworks like FastAPI, which uses it extensively for request and response validation.
Ease of Use: Leverages Python type hints, reducing the need for verbose validation logic.

Code Example

Here’s a simple example to demonstrate Pydantic in action:

from pydantic import BaseModel, ValidationError
from typing import List

class User(BaseModel):
    id: int
    name: str
    is_active: bool = True
    tags: List[str] = []

# Valid input
user_data = {
    'id': '123',  # Will be coerced to int
    'name': 'Alice',
    'tags': ['python', 'developer']
}

user = User(**user_data)
print(user)
print(user.model_dump())

# Invalid input
try:
    invalid_data = {'id': 'abc', 'name': 234}
    User(**invalid_data)
except ValidationError as e:
    print(e)

Output:

id=123 name='Alice' is_active=True tags=['python', 'developer']
{'id': 123, 'name': 'Alice', 'is_active': True, 'tags': ['python', 'developer']}
2 validation errors for User
id
  Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='abc', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/int_parsing
name
  Input should be a valid string [type=string_type, input_value=234, input_type=int]
    For further information visit https://errors.pydantic.dev/2.11/v/string_type

While both Pydantic and TypedDict from the typing module allow type annotations for dictionaries, they serve different purposes:

Feature	Pydantic	TypedDict
Validation	Yes	No
Default Values	Yes	Yes (in Python 3.9+)
Type Coercion	Yes (e.g., str → int)	No
Runtime Behavior	Full-featured at runtime	Mainly for static analysis
Error Reporting	Detailed errors via exceptions	None
Performance	Slightly slower due to checks	Faster (no runtime overhead)

Summary: Use TypedDict when you only need type hints for static checking (e.g., with mypy). Use Pydantic when you need runtime validation, coercion, and structured error handling.

Conclusion

Pydantic is a powerful and user-friendly library that brings the type-safe validation into dynamic Python code. It helps developers write cleaner, more maintainable code by enforcing data structure constraints and providing detailed feedback on invalid inputs. Whether you're building a REST API, working with external data sources, or managing configurations, Pydantic can be an essential part of your Python toolkit.