Forem: Suave Bajaj

Stop Getting KeyError: '\ufeffName' When Reading CSVs in Python — Here’s the Fix

Suave Bajaj — Sun, 30 Nov 2025 12:27:05 +0000

If you’ve ever tried reading a CSV file in Python and suddenly got this confusing error:

KeyError: '\ufeffName'

…you’re not alone.

This problem happens even when the CSV looks perfectly fine, especially if it was exported from Windows or Excel.
There’s an invisible character at the start of the file that Python reads — but you don’t.

Let’s break down the issue and the one-line fix.

The Hidden Problem: BOM (\ufeff)

Many CSV files created on Windows tools include a Byte Order Mark (BOM).

It’s invisible, but Python treats it as part of the header.

Example Problem:

CSV file (data.csv):

Name,Age,Email
Alice,25,alice@example.com
Bob,30,bob@example.com
Charlie,28,charlie@example.com

If this CSV has a BOM, Python might read the first header as

\ufeffName

instead of Name, causing key errors:

import csv

with open("data.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        print(row["Name"])

Output:

KeyError: '\ufeffName'

The One-Line Fix: Use utf-8-sig

Python has a specific encoding that automatically removes the BOM while reading:

utf-8-sig

Just open your file like this:

import csv

with open("data.csv", "r", encoding="utf-8-sig") as file:
    reader = csv.DictReader(file)
    for row in reader:
        print(row["Name"], row["Age"], row["Email"])

That’s it. Problem solved.

Output:

Alice 25 alice@example.com
Bob 30 bob@example.com
Charlie 28 charlie@example.com

Why This Works

CSVs exported from Excel often include a BOM

The BOM is invisible but breaks Python’s header parsing utf-8-sig automatically removes the BOM before DictReader sees it

Your script works with any CSV source — Linux, Windows, Mac

When Should You Use utf-8-sig?

Use it whenever your CSV comes from:

Excel
Windows-based exports
Enterprise systems
Unknown or mixed data sources

It’s a safe default and won’t break anything even if the file contains no BOM.

Summary

If you encounter weird CSV header issues or errors like:

KeyError: '\ufeffName'

just open your CSV like this:

open("file.csv", "r", encoding="utf-8-sig")

This one line removes BOM automatically and makes your CSV reading code clean, stable, and cross-platform.

Kubernetes Autoscaling with Custom Scaler: Event-Driven Scaling for Queues and Microservices – Part 2

Suave Bajaj — Fri, 26 Sep 2025 10:00:04 +0000

In Part 1, we explored KEDA and how it scales your consumers based on queue depth.

But what if:

You have N queues → M consumers
Each queue has different thresholds, min, and max replicas
Each consumer has a different workflow / endpoint

KEDA alone can’t handle this. That’s where a custom autoscaler comes in.

Problem Statement

Example scenario:

Queue	Consumer Deployment	Threshold	Min Pods	Max Pods
x-queue-1	consumer-x-1	100	1	5
x-queue-2	consumer-x-2	50	2	6
y-queue	consumer-y	200	1	10

Each queue has its own processing logic.
Scaling decisions must be independent.
Min/Max replicas can be defined dynamically in a database for flexibility.

Architecture Overview

┌───────────────────────┐
│     Producers          │
│  (Apps push messages   │
│   to queues)           │
└─────────┬─────────────┘
          │
          ▼
 ┌─────────────────┐
 │  Message Broker  │
 │  (Kafka/Rabbit)  │
 └─────────┬────────┘
           │
┌──────────┴──────────┐
▼                     ▼
┌─────────────────────┐       ┌─────────────────────┐
│  Python Exporter    │       │ Prometheus Metrics  │
│ - Reads queue depth │◄─────►│ storage             │
│ - Reads min/max     │       └─────────────────────┘
│   from DB           │
│ - Applies scaling   │
│   logic per queue   │
│ - Calls Kubernetes  │
│   API to scale      │
└─────────┬───────────┘
          │
          ▼
┌─────────────────────────────┐
│ Consumer Deployments         │
│ - pod-deployment-1          │
│ - pod-deployment-2          │
│ - pod-deployment-y          │
└─────────────────────────────┘

Python Example: Custom Scaling with DB Min/Max

from kubernetes import client, config
import requests
import sqlite3  # Example DB; replace with your real DB

# Load in-cluster config
config.load_incluster_config()
apps_v1 = client.AppsV1Api()

# Connect to database containing min/max per consumer
conn = sqlite3.connect('consumer_scaling.db')
cursor = conn.cursor()

# Read scaling config for all consumers
cursor.execute("SELECT consumer_name, queue_name, threshold, min_pods, max_pods FROM scaling_config")
scaling_rules = cursor.fetchall()

# Prometheus endpoint
prometheus_url = "http://prometheus:9090/api/v1/query"

for consumer_name, queue_name, threshold, min_pods, max_pods in queue_config:
    # Query current queue depth
    query = f'sum(queue_depth{{queue="{queue_name}"}})'
    resp = requests.get(prometheus_url, params={"query": query}).json()
    queue_depth = float(resp["data"]["result"][0]["value"][1])

    # Calculate desired replicas using custom logic
    desired_replicas = max(min_pods, min(max_pods, int(queue_depth / threshold)))

    # Scale Deployment
    scale = client.V1Scale(spec=client.V1ScaleSpec(replicas=desired_replicas))
    apps_v1.replace_namespaced_deployment_scale(
        name=consumer_name,
        namespace="default",
        body=scale
    )

    print(f"{consumer_name}: queue={queue_depth}, threshold={threshold}, scaled to {desired_replicas} pods")

Notes

threshold is per queue.
min_pods and max_pods are read from a database, making it dynamic.
You can extend logic to include weighted scaling, multiple metrics, or cooldown periods.

Advantages of Custom Autoscaler

Independent scaling per queue
Dynamic min/max replicas from DB — no hardcoding
Multi-metric decisions (queue depth, CPU, DB lag, etc.)
Advanced logic (cooldowns, weighted scaling, prioritization)
Can handle N queues → M consumers mapping flexibly

Trade-offs

Feature	KEDA	Custom Autoscaler
Easy setup	✅	❌ (Python + DB + K8s API)
Independent queue scaling	❌	✅
Multi-metric logic	Limited	✅
DB-driven min/max	❌	✅
Reliability	✅ Battle-tested	⚠️ Managed by you

✅ Takeaways

KEDA is great for simple queue scaling.
For complex microservices with multiple queues, custom autoscaler gives full control.
Using a database for min/max replicas allows dynamic, production-ready scaling policies.
Your custom autoscaler can evolve into a custom HPA/KEDA tailored to your architecture.

💡 Pro Tip:

Start with KEDA for simple cases. Move to a custom autoscaler with DB-defined min/max for multi-queue microservices with complex workflows.

Kubernetes Autoscaling with KEDA: Event-Driven Scaling for Queues and Microservices – Part 1

Suave Bajaj — Sat, 20 Sep 2025 09:23:33 +0000

KEDA & Event-Driven Autoscaling: Solving Queue Backlogs in Kubernetes

If you've ever run Kafka or RabbitMQ consumers in Kubernetes, you know the pain: your consumers are either underutilized or overloaded, and CPU-based HPA just doesn’t cut it.

Enter KEDA — Kubernetes-based Event-Driven Autoscaling.
KEDA Github

What is KEDA?

KEDA is a lightweight autoscaler that scales your Kubernetes workloads based on external events or metrics, such as:

Queue length in Kafka or RabbitMQ
Prometheus metrics
Cloud events (Azure, AWS, GCP)

Unlike HPA which scales on CPU/memory, KEDA lets your pods scale dynamically based on actual workload.

Important: KEDA itself runs as a Deployment in your Kubernetes cluster.

It’s not a StatefulSet or a special Deployment of your app.

It consumes CPU/memory like any other Deployment depending on the number of ScaledObjects and triggers you define.

This means if you define many queues, KEDA’s own Deployment will grow in resource usage.

How KEDA Works

KEDA introduces a new Kubernetes object called a ScaledObject:

ScaledObject → Deployment mapping: tells KEDA which workload to scale.
Each ScaledObject can have one or more triggers.
Scaling logic:
- OR logic across triggers (if any trigger exceeds threshold → scale).
- Cannot independently scale subsets of pods within the same Deployment.

Example: RabbitMQ ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: orders-consumer-scaler
spec:
  scaleTargetRef:
    name: orders-consumer
  minReplicaCount: 1
  maxReplicaCount: 10
  triggers:
  - type: rabbitmq
    metadata:
      queueName: orders-queue
      host: amqp://guest:guest@rabbitmq:5672/
      queueLength: "100"

Explanation:

orders-consumer Deployment will scale 1→10 pods depending on queue length.
Scaling happens if the queue length > 100.

KEDA & Event-Driven Autoscaling: Solving Queue Backlogs in Kubernetes

If you've ever run Kafka or RabbitMQ consumers in Kubernetes, you know the pain: your consumers are either underutilized or overloaded, and CPU-based HPA just doesn’t cut it.

Enter KEDA — Kubernetes-based Event-Driven Autoscaling.

How KEDA Determines the Number of Pods

KEDA computes desired replicas based on the queue metric and thresholds:

Threshold value (queueLength)
- The metric KEDA monitors (queue depth).
Min and Max Replicas

minReplicaCount: 1
maxReplicaCount: 10

**Desired replicas calculation (simplified):

desiredReplicas = ceil(currentQueueLength / queueLengthThreshold)

**Scaling applied via HPA

KEDA updates the Deployment replicas between minReplicaCount and maxReplicaCount

Visual Example

Imagine orders-queue has a threshold of 100 messages:

Queue Depth → Desired Replicas → Deployment Pods
------------------------------------------------
50          → 1                 → 1 pod (minReplicaCount)
120         → 2                 → 2 pods
250         → 3                 → 3 pods
1050        → 11                → 10 pods (maxReplicaCount)

Diagram: Queue Depth → Desired Replicas → Deployment Scaling

[Queue Depth: 250]
        │
        ▼
[Threshold: 100 per pod]
        │
        ▼
[Desired Replicas: ceil(250/100) = 3]
        │
        ▼
[orders-consumer Deployment scales to 3 pods]

Limitations of KEDA

While KEDA is powerful, it has some critical limitations:

N queues → N consumers is not supported natively

You cannot define multiple queues with different consumer Deployments in a single ScaledObject.

Scaling logic is OR across triggers; all pods scale together.

Not ideal for microservices with multiple queues and workflows

Example: if you have 10 queues with different processing logic/endpoints, KEDA alone cannot scale them independently.

You’d need multiple Deployments + multiple ScaledObjects, which becomes cumbersome.

Resource consumption

KEDA itself is a Deployment; each ScaledObject adds CPU/memory usage.

Scaling many queues can increase the resource footprint on the cluster.

Kubectl Example

After creating multiple ScaledObjects:

kubectl get scaledobjects

Example output:

NAME                       SCALETARGETKIND      SCALETARGETNAME     MIN   MAX   TRIGGERS
orders-consumer-scaler     apps/v1.Deployment   orders-consumer      1    10    rabbitmq
payments-consumer-scaler   apps/v1.Deployment   payments-consumer    1     5    rabbitmq
kafka-consumer-scaler      apps/v1.Deployment   kafka-consumer       1    15    kafka

Each ScaledObject targets a single Deployment.

Multiple triggers for the same Deployment scale all pods together (OR logic).

Takeaways

KEDA is great for simple event-driven scaling.
Each ScaledObject → one Deployment.
OR logic for multiple triggers → not suitable for microservices with multiple queues and workflows.
KEDA computes replicas automatically using the metric and threshold, scaling Deployment pods between min and max.
Resource footprint increases with more ScaledObjects and triggers.

Part 2 will show how to build a custom autoscaler that can handle multiple queues, multiple consumers, and independent scaling logic for complex microservices.

Pro Tip: Start with KEDA for simple queue scaling. For complex, multi-queue microservices, a custom HPA/KEDA solution is usually required.

Docker Best Practices: Reduce Image Size + Common Interview Questions

Suave Bajaj — Wed, 10 Sep 2025 09:43:26 +0000

When working with Docker images in production, size matters a lot.

Large images mean longer build times, slower deployments, and even higher storage costs in container registries like GCP Artifact Registry or Docker Hub.

In one of our projects, the image size grew beyond 1.9 GB, and we optimized it down to just 495 MB — a 75% reduction. Here’s how we did it.

Bonus : Have shared common inteview prep questions and answers in the end.

🔹 Why Docker Image Size Matters

⏱ Faster builds & CI/CD pipelines
📦 Less storage usage in registries
🚀 Faster deployments & scaling
💰 Lower cloud costs
🔒 Smaller attack surface

🔹 Our Starting Point

We started with this basic Dockerfile:

FROM google/cloud-sdk:latest

WORKDIR /app
COPY . .
RUN pip install -r requirements.txt

CMD ["python3", "app.py"]

❌ Problems with this approach

google/cloud-sdk:latest is ~3 GB 😱
Copying everything at once invalidates Docker cache → slow builds
No .dockerignore → unnecessary files (e.g., .git, __pycache__, .vscode/) got copied
pip install cached dependencies → bloated layers

🔹 Optimized Approach

We switched to a smaller base image and installed only what we need.

✅ Optimized Dockerfile


FROM python:3.12-slim

WORKDIR /app

# Install gcloud SDK (instead of using 3GB google/cloud-sdk image)
RUN apt-get update && \
    apt-get install -y apt-transport-https ca-certificates gnupg curl && \
    echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
    curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg && \
    apt-get update && \
    apt-get install -y google-cloud-sdk && \
    rm -rf /var/lib/apt/lists/*

# Copy only requirements first (for caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

CMD ["python3", "app.py"]

🔹 Why This Works

Slim Base Image
- python:3.12-slim is just 124 MB vs google/cloud-sdk:latest at ~3 GB.
- We add only the gcloud tools we need (gsutil, gcloud sql, etc.).
Cache-Friendly Layering
- COPY requirements.txt first → pip install is cached until dependencies change.
- App code changes don’t trigger dependency reinstall.
.dockerignore Optimization

A .dockerignore file prevents unnecessary files from being copied into the image:

   __pycache__/
   *.pyc
   *.pyo
   *.pyd
   .Python
   .git
   .vscode
   .DS_Store

This keeps unnecessary files out of the image.

No Pip Cache
pip install --no-cache-dir avoids leaving hundreds of MB in cached wheels.
APT Cleanup
rm -rf /var/lib/apt/lists/* keeps layers lean.

📊 Before vs After

Version	Size	Notes
`testv1.5.0`	1.96 GB	Full `google/cloud-sdk`, no cleanup
`v1.6.9`	1.36 GB	Slim base, partial optimization
`testv1.4.1`	495 MB	Slim base + caching + `.dockerignore` + cleanup

💡 Result: 75% reduction in size 🚀

3x faster builds
Faster pushes/pulls from Artifact Registry
Lower cloud storage costs

🔹 Common Interview Questions on Docker Image Optimization

If you’re preparing for DevOps/SRE interviews, expect questions like:

Q1. How do you reduce Docker image size?

Use slim base images.
Multi-stage builds.
.dockerignore.
Remove build-time dependencies.
Clean caches (pip, apt).

Q2. Why copy requirements.txt separately before app code?

To leverage Docker’s layer caching → only reinstall deps when requirements change.

Q3. What’s the role of .dockerignore?

Prevents unnecessary files (like .git, logs, caches) from being copied → smaller, cleaner image.

Q4. When should you use multi-stage builds?

When compiling dependencies (Go, Java, C extensions).
First stage: build artifacts.
Second stage: copy only binaries/libs into a minimal runtime image.

Q5. Why not use google/cloud-sdk:latest directly?

It’s huge (~3 GB).
Installing only required gcloud components on python:slim keeps image much smaller.

🚀 Final Thoughts

Optimizing Dockerfiles is not just about saving space — it impacts CI/CD speed, developer productivity, and cloud costs.

By combining slim images, layer caching, .dockerignore, and minimal installs, you can cut image sizes dramatically.

[Boost]

Suave Bajaj — Sun, 20 Jul 2025 08:09:35 +0000

Suave Bajaj

Jul 15 '25

Authenticate Docker with Google Artifact Registry (Private Repo) Using a Service Account

#docker #gcp #devops #kubernetes

Comments

2 min read

Authenticate Docker with Google Artifact Registry (Private Repo) Using a Service Account

Suave Bajaj — Tue, 15 Jul 2025 08:03:32 +0000

⚡ TL;DR: Want to push/pull private Docker images from Google Artifact Registry? Use a Google Cloud service account with docker login -u _json_key, then use the .docker/config.json for Kubernetes integration.

🧰 Prerequisites

Make sure you have the following:

Docker installed locally or in your CI/CD agent
Google Cloud SDK (gcloud)
A private Artifact Registry (e.g. us-central1-docker.pkg.dev/<project>/<repo>)
A service account with Artifact Registry Reader or Writer permissions

⚙️ Step 1: Create a Service Account Key

Create a key file to authenticate Docker later:

# Create a new key for your artifact registry service account
gcloud iam service-accounts keys create key.json \
  --iam-account=gar-access@<PROJECT_ID>.iam.gserviceaccount.com

This will download a key.json file locally.

🔐 Step 2: Authenticate Docker with the Key

Use the service account to log in to your private Artifact Registry:

# Replace the region with your registry's location
cat key.json | docker login -u _json_key --password-stdin https://us-central1-docker.pkg.dev

This command updates your ~/.docker/config.json to include authentication for the private registry.

📁 What the Docker Config File Looks Like

After successful login, your ~/.docker/config.json will look like this:

{
  "auths": {
    "https://us-central1-docker.pkg.dev": {
      "auth": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9..."  // Base64 token
    }
  },
"currentContext": "colima" // As per your docker client
}

You can use this file:

As a Kubernetes image pull secret
In CI/CD pipelines
For temporary access without gcloud

🚀 Use in CI/CD and Kubernetes

🧪 In CI/CD (GitHub Actions, GitLab, Jenkins, etc.)

You can use the same docker login step using the service account key during your pipeline execution.

📦 In Kubernetes

To use this config as a Kubernetes secret:

kubectl create secret generic regcred \
  --from-file=.dockerconfigjson=$HOME/.docker/config.json \
  --type=kubernetes.io/dockerconfigjson

Reference it in your Deployment:

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: regcred

🔍 Verify the Setup

Once authenticated, verify the setup by building, tagging, and pushing an image to your private Artifact Registry.

🛠️ Build the Docker Image

docker build -t myapp:latest .
#Replace myapp with your actual app name, and ensure your Dockerfile is in the current directory.

🏷️ Tag the Image

docker tag myapp:latest us-central1-docker.pkg.dev/<project>/<repo>/myapp:latest
#Replace <project> and <repo> with your actual GCP project ID and Artifact Registry repository.

📤 Push the Image

docker push us-central1-docker.pkg.dev/<project>/<repo>/myapp:latest

📥 Optionally, Pull to Confirm

docker pull us-central1-docker.pkg.dev/<project>/<repo>/myapp:latest

If these steps complete without errors, your service account authentication and private registry access are working perfectly!

✅ Summary and References

Use a Google Cloud service account with _json_key to securely access private Artifact Registry images.
docker login will generate a valid config.json.
Use that config for Kubernetes and CI/CD integration.
No need for gcloud runtime dependencies in CI.

📘 Google Artifact Registry Authentication
🐳 Docker Credential Helpers
☸️ Kubernetes: Pull an Image from a Private Registry

🚀 My First Flask API – Lightweight Monitoring Data Server (with Deep Dive)

Suave Bajaj — Wed, 09 Jul 2025 07:11:28 +0000

Hey folks! 👋

I recently built my first API using Flask and wanted to share the experience — not just the code, but also how it works, what each part does, and how you can run it yourself. If you're learning Flask or curious about simple data-serving APIs, this might help!

🧠 What This API Does

This Flask API serves precomputed monitoring data for various backend services (like Kafka, RabbitMQ, etc.) using .json files. Here's what it can do:

✅ 1. Returns JSON Files Containing Health Data

Data for each service is stored as JSON files in a structured directory.
You can query the API with parameters like stackname, service, and time to get specific metrics.
Example:

/getdata?stackname=retail&service=kafka&time=20240708

This returns the contents of:

`/data-files/retail/kafka_20240708.json`

✅ 2. Compares Two Time Periods for the Same Service

You can compare a service's performance between two timestamps:

/getdata/comparison?stackname=retail&service=kafka&start_time=20240701&end_time=20240707

This returns:

/data-files/retail/comparisons/kafka_20240701_vs_20240707.json

Useful for catching trends, regressions, or improvements.

✅ 3. Displays a Landing Page with All Available Endpoints

Visit the root / to get a helpful summary:

{
"message": "👋 Welcome to the API!",
"available_endpoints": {
  "/getdata/comparison": "GET with params: stackname, service, start_time, end_time",
  "/getdata": "GET with params: stackname, service, time"
},
"available_services": [
  "elasticsearch", "rabbitmq", "pods", "responsetime",
  "logsearch", "kafka", "database", "asciapp", "consul"
],
"status": "running"
}

🔍 Code Walkthrough

app.py

from flask import Flask
from datahandler import DataHandler

handler = DataHandler()
app = Flask(__name__)

app.add_url_rule('/getdata/comparison', 'getcomparisondata', handler.get_comparison_data)
app.add_url_rule('/getdata', 'stackdata', handler.get_stack_data)
app.add_url_rule('/', 'landingpage', handler.get_landing_page)

if __name__ == '__main__':
    app.run(debug=True, port=8080)

datahandler.py

from flask import request, send_file, abort, Response, json
import os

BASE_DIR = os.path.dirname(os.path.abspath(__file__))
files_file_path = os.path.abspath(os.path.join(BASE_DIR, "..", "data-files"))
config_path = os.path.abspath(os.path.join(BASE_DIR, "..", "config", "data_config.json"))

with open(config_path) as f:
    config = json.load(f)

class DataHandler:
    valid_data_paths = config.get("valid_data_paths", [])

    def get_comparison_data(self):
        stackname = request.args.get('stackname')
        service = request.args.get('service')
        start_time = request.args.get('start_time')
        end_time = request.args.get('end_time')
        file_name = f"{stackname}_{service}_{start_time}_vs_{end_time}.json"
        path = os.path.join(files_file_path, stackname, "comparisons", file_name)
        return send_file(path, mimetype='application/json')

    def get_stack_data(self):
        stackname = request.args.get('stackname')
        service = request.args.get('service')
        time = request.args.get('time')
        if service in self.valid_data_paths:
            file_name = f"{stackname}_{service}_{time}.json"
            path = os.path.join(files_file_path, stackname, file_name)
            return send_file(path, mimetype='application/json')
        return abort(400, description="Invalid data type")

    def get_landing_page(self):
        data = {
            "message": "Welcome to the API!",
            "available_endpoints": {
                "/getdata/comparison": "GET with params: stackname, service, start_time, end_time",
                "/getdata": "GET with params: stackname, service, time"
            },
            "available_services": self.valid_data_paths,
            "status": "running"
        }
        return Response(json.dumps(data, indent=2), mimetype='application/json')

▶️ How to Run It

Install Flask:

pip install flask

Run the python code

python app.py

Open your browser and visit or use the POSTMAN tool to get the details

http://localhost:8080/

💡 What Does Each Line in app.run() Do?

app.run(debug=True, port=8080)

Argument	What It Does
`debug=True`	Auto-reloads the app on code changes and shows detailed error messages. Great for dev, not for prod.
`port=8080`	Runs the app on port 8080 instead of the default 5000.
`host='0.0.0.0'` (optional)	Allows access from any IP address (e.g., Docker or public cloud).

app.run(host='0.0.0.0', port=8080, debug=True)

🔗 Query Parameters in Flask

You pass arguments like this:

/getdata?stackname=prod&service=pods&time=0708

And extract them in code:

request.args.get('stackname')
request.args.get('service')
request.args.get('time')

You can also give default values:

request.args.get('service', 'default_service')

🛠️ Lessons I Learned

How to use Flask routing and send_file() for static JSON
The importance of modular design (datahandler.py)
Building an API without needing a full DB
Making endpoints self-documenting for easy testing and consumption

Capture the Magic: Mastering Star Trails with Your Galaxy S23 Ultra

Suave Bajaj — Mon, 24 Jun 2024 08:51:51 +0000

Have you ever gazed up at a night sky teeming with stars and wished you could capture their mesmerizing movement? With the innovative camera features of the Galaxy S23 Ultra, transforming that starry expanse into a captivating image of star trails is within reach!

This guide will unveil the secrets to capturing stunning star trails using your S23 Ultra, allowing you to immortalize the Earth's rotation on camera. Watch the youtube short of the star trails here

Gear Up for the Night:

Galaxy S23 Ultra: The star of the show, of course!
Tripod: Essential for ensuring complete image stability during the long exposure required for star trails.
Remote Shutter Release (Optional): Minimizes camera shake when triggering the capture.

Finding the Perfect Night Sky:

Escape Light Pollution: City lights wash out the faint starlight needed for star trails. Seek out a dark location, preferably far from urban areas. National parks and remote areas are ideal.
Favor Clear Skies: Obstructions like clouds can significantly hinder your capture. Aim for a clear, starry night with minimal cloud cover.

Unleashing the Power of Hyperlapse:

The Galaxy S23 Ultra's Hyperlapse mode is the key to creating stunning star trails. Here's how to use it:

Launch the Camera App: Open the default camera application on your S23 Ultra.
Switch to Hyperlapse Mode: Swipe through the shooting modes until you find "Hyperlapse."
Maximize Resolution: Tap on the resolution icon and choose the highest option, ideally UHD (Ultra High Definition).
Set the Speed: Select the recording speed. For capturing star trails, a speed of X300 is recommended.
Activate Star Trails: Look for the "Star Trails" icon (it might resemble tiny stars) and tap on it to enable this specific capture mode within Hyperlapse.
Frame Your Shot: Compose your desired shot. Consider including interesting foreground elements like mountains or trees to add depth to your final image.

Capturing the Night's Journey:

Hit Record: Press the shutter button to begin recording the star trails. Remember, capturing good star trails requires a long exposure, so be prepared to let the camera record for at least 1 hour to 2 hours (or even longer) for a more dramatic effect. Watch the youtube short of the star trails here
Maintain Stability: Using a tripod is crucial. Even the slightest movement can blur your image. A remote shutter release can further minimize camera shake.
Monitor Battery Life: Recording for extended periods can drain your battery. Ensure you have a power bank or a way to keep your phone charged throughout the capture.

Post-Capture Magic:

Once the recording is complete, the S23 Ultra will process the Hyperlapse footage and create a stunning video showcasing the movement of the stars across the night sky. You can further enhance your Star Trails video using editing software, adjusting brightness, contrast, and saturation to taste.

Bonus Tip: Download the Samsung Expert RAW app to unlock even more control over your astrophotography experience.

With these steps and a touch of starry-eyed wonder, you'll be well on your way to capturing breathtaking star trails with your Galaxy S23 Ultra, transforming fleeting moments into mesmerizing celestial art!
Watch the youtube short of the star trails here

ZooKeeper Chronicles: Navigating EOFException and the Enigma of 0-Length Files

Suave Bajaj — Mon, 18 Dec 2023 18:55:58 +0000

In the realm of technology, ZooKeeper plays a crucial role in maintaining order within distributed systems. However, much like any compelling narrative, I recently encountered a stumbling block marked by an unexpected challenge – an EOFException paired with an intriguing 0-length file.

The Error:

ERROR [main:Util@214] - Last transaction was partial.
ERROR [main:ZooKeeperServerMain@66] - Unexpected exception, exiting abnormally
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)

An EOFException occurred, causing our ZooKeeper server to crash.

The Mysterious 0 Length File:

Navigating through the ZooKeeper folder, I saw something strange:

-rw-rw-r-- 1 zookeeper zookeeper 1342177280 Sep 23 23:17 log.be45c2
-rw-rw-r-- 1 zookeeper zookeeper          0 Oct  1 14:31 log.be85e9

Digging Deeper:

The zero file hinted at trouble with our transaction logs. Looking into the logs, we found the last transaction was a bit messed up, triggering the EOFException. But why did it result in a zero file?

Real Issue: Running Out of Space

There was a 0 Length File because the underlying disk utilization was full and no space was left on the device to write the logs.

Filesystem      Size  Used Avail Use% Mounted on
/dev/sde         25G   25G     0 100% /var/lib/zookeeper

As per the Zookeeper official documentation here

A ZooKeeper server will not remove old snapshots and log files, this is the responsibility of the operator. Every serving environment is different and therefore the requirements of managing these files may differ from install to install (backup for example).

The PurgeTxnLog utility implements a simple retention policy that administrators can use. The API docs contains details on calling conventions (arguments, etc...).

Using the PurgeTxnLog utility

zookeeper@zk-0:/bin$ zkCleanup.sh
Usage:
PurgeTxnLog dataLogDir [snapDir] -n count
    dataLogDir -- path to the txn log directory
    snapDir -- path to the snapshot directory
    count -- the number of old snaps/logs you want to keep, value should be greater than or equal to 3

Keep the latest 5 logs and snapshots
./zkCleanup.sh -n 5

Conclusion:

In the dynamic landscape of distributed systems, unexpected errors often hint at underlying complexities. Our journey through the EOFException and zero file quandary emphasizes the importance of vigilant monitoring of logs and disk space. As tech custodians, we must be prepared to decode mysteries, address core issues, and ensure the seamless operation of systems like ZooKeeper.

ZooKeeper, with its unique quirks and challenges, continues to be an exciting playground for tech enthusiasts eager to navigate the intricacies of system synchronization.

Taint Like a Pro: Effortlessly Taint Multiple Nodes Using Node Labels in Kubernetes!

Suave Bajaj — Mon, 31 Jul 2023 07:58:32 +0000

Introduction:

In Kubernetes, taints and tolerations offer a powerful mechanism for controlling pod placement on nodes. Taints enable nodes to repel pods, ensuring they only accept those with matching tolerations. However, manually tainting individual nodes can be time-consuming. In this guide, we’ll explore a more efficient approach by using node labels to taint multiple nodes simultaneously.

Understanding Taints and Tolerations:

Taints are used to mark nodes, indicating that they should not accept pods that lack specific tolerations. Tolerations, on the other hand, allow pods to be scheduled on tainted nodes. Together, they provide fine-grained control over pod placement based on node characteristics.

Tainting a Node:

To taint a node, use the kubectl taint command followed by the node name and the taint specification.

For instance:

kubectl taint nodes node1 key1=value1:NoSchedule

This example taints the node named node1 with the key-value pair key1=value1 and the NoSchedule effect. Consequently, no pod without a matching toleration will be scheduled onto node1.

Get the Tainted Nodes:

To verify which nodes have taints, you can use the kubectl get nodes command with the JSON output format and filter it using jq:

kubectl get nodes -o json | jq ".items[] | {name:.metadata.name, taints:.spec.taints}"

This command provides a clear view of the nodes along with their corresponding taints.

Tainting Multiple Nodes with Node Labels:

When you have a group of nodes with similar attributes (e.g., identical machine types or workloads), tainting them individually becomes impractical. Instead, you can utilize node labels to taint all nodes matching a specific criterion in one go.

kubectl taint nodes -l label_key=label_value tainting_key=tainting_value:NoSchedule

Example:

kubectl taint nodes -l machine-type=e2-small machine-type=e2-small:NoSchedule

In this example, all nodes labelled machine-type=e2-small will be tainted with the NoSchedule effect. Consequently, only pods with tolerations matching the specified taint will be scheduled on these nodes.

Advantages of Tainting with Node Labels:

Time-saving: By tainting multiple nodes simultaneously using labels, you avoid repetitive manual operations, reducing the time and effort required for node management.
Workload migration: Node tainting with labels facilitates seamless workload migration from one group of nodes to another, based on specific criteria.
Resource optimization: Efficiently balance workloads across different node specifications, ensuring optimal resource utilization throughout your Kubernetes cluster.

Conclusion:
Using node labels to taint multiple nodes in Kubernetes is a powerful technique that simplifies node affinity management. By applying taints based on labels, you gain greater control over pod placement and workload distribution, streamlining operations and optimizing resource usage. Embrace this approach to efficiently manage your Kubernetes cluster and ensure smooth workload migration across nodes with diverse characteristics.

Create Signed URLs using gsutil

Suave Bajaj — Thu, 24 Mar 2022 10:03:02 +0000

A signed URL is a URL that provides limited permission and time to make a request. Signed URLs contain authentication information in their query string, allowing users without credentials to perform specific actions on a resource.

Creating a signed URL

What we will need:

gsutil
pyopenssl
Service Account Key
Path of the file/Object

Command

gsutil signurl -d 2d service-account-key.json gs://my-gcs-bucket/my-object

-d Specifies the duration that the signed url should be valid
For, default duration is 1 hour.Times may be specified with no suffix (default hours), or with s = seconds, m = minutes, h = hours, d = days.
This option may be specified multiple times, in which case
the duration the link remains valid is the sum of all the duration options.
The max duration allowed is 7 days when private-key-fileis used.
Any service-account-key can be used to encrypt the url

Issues

CommandException: The signurl command requires the pyopenssl library (try pip install pyopenssl or easy_install pyopenssl)

Install pyopenssl using the pip as per the python version

pip3 install pyopenssl

Even after installing pyopenssl if you are getting the same error, Check gcloud uses which python and python should point to a python interpreter used by gcloud

gcloud info --format="text(basic.python_location)"

➜ Desktop git:(master) ✗ gcloud info --format="text(basic.python_location)" python_location: /Users/.config/gcloud/virtenv/bin/python3

Add the Environment Variables as

export CLOUDSDK_PYTHON=python3
export CLOUDSDK_PYTHON_SITEPACKAGES=1

Make Elasticsearch 6.x Cluster Writable from Readonly

Suave Bajaj — Tue, 08 Mar 2022 10:35:18 +0000

Sometimes you might get

Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

This error indicates a data node is critically low on disk space and has reached the flood-stage disk usage watermark. To prevent a full disk, when a node reaches this watermark, Elasticsearch blocks writes to any index with a shard on the node. If the block affects related system indices, Kibana and other Elastic Stack features may become unavailable.

This means you'll only be able to read the Indices, Delete the Indices but cannot write new Indices.

Get the Cluster Settings

curl elasticsearch-sc:9200/_settings?pretty

You'll see the read_only_allow_delete following as true for the Indices that means the cluster is in read-only mode

"blocks" : { "read_only_allow_delete" : "true" },

After the Disk Cleanup, When you have deleted the non-required Indices or after adding more space to the cluster, The Cluster needs to be put back to the writable mode

Putting Cluster back to Writable Mode

curl -XPUT -H "Content-Type: application/json" \ http://elasticsearch-sc:9200/_all/_settings \ -d '{"index.blocks.read_only_allow_delete": null}'

You can check the cluster settings again to confirm the change