Forem: Akash Singh

DO NOT AUTOSCALE : PBCTF RCA

Akash Singh — Wed, 24 Sep 2025 17:35:11 +0000

The Incident

On 02.08.25, during PBCTF 4.0 - our college club Point Blank's flagship CTF competition - we experienced what every CTF organizer fears: a complete platform outage. For 35 agonizing minutes from 10:00AM to 10:35AM IST, Me and Govind tried bringing the service back up while 45-50 of our other teammates made memes and helped calm the revolting participants.

The incident was not a complete outage as we had a backup ctfd deployment running on our own PB Server, pointing to a different DNS, to which we directed traffic. Sadly, the PB Server was not powerful enough to handle all 400 participants but it did take care of approximately half of them while the rest of PB Members distracted the other half set of participants.

Credits to our senior Ashutosh Pandey for the idea to keep a backup deployment ready.

Introduction

I am Akash Singh, a final year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

Timeline of Events

T-0: The Calm Before the Storm

The opening ceremony went great and we were ready to get the CTF Started. Both deployments were healthy and participants were logging on and getting ready for the CTF.

Our GCP Kubernetes cluster was a sweet deal, we spent 1000 INR to get 25000 INR in GCP credits and felt invincible with all that cloud power at our disposal.

T+5 minutes: "Let's Scale!"

Traffic was picking up. In a moment of what seemed like devops big brain time (it wasn't), I decided to scale up our pods. After all, who doesn't love autoscaling? 25000 INR needs to be used afterall.

# What could possibly go wrong?
spec:
  replicas: 2  # YOLO

Me Chilling after ""AutoSCAllINg""

T+10 minutes: The First Signs of Trouble

New pods started spinning up. They attempted to run database migrations - standard procedure, right? Wrong. The pods immediately crashed with migration errors.

T+15 minutes: Panic Mode Activated

Our initial hypothesis: "Must be a migration issue!"

The night before, Govind said he ran some custom migrations. Maybe something was incompatible? In my big brain, I decided to delete all pods and start fresh.

kubectl delete pods --all -n ctfd
# Famous last words: "This will fix it"

T+16 minutes: Full Outage

Congratulations, we played ourselves. All pods gone. CTFd completely down on k8s deployment and then all of PB started calling at the same time :

The Backup That Saved Us

Here's where we owe a massive thank you to Ashutosh bhaiya for his prescient advice: "Always keep a backup deployment ready."

We had a secondary CTFd instance running on our PB server - completely separate from the Kubernetes cluster but connected to the same database. While Govind and I battled with the K8s deployment, this backup instance kept serving traffic to some participants.

The Real Culprit

After 30 minutes of debugging, checking migration scripts and nearly crying to claude for a fix, I saw one emergency pod created after some random claude commands finally healthy.

After everything settled down and we sat down for the RCA, we found out the real issue.

"We were using two different versions of CTFd from two different servers"

Lavi's deployment : On PB Server: Latest CTFd version
Govind's deployment : On GCP K8S: CTFd 3.7.2

Both connecting to the same database. Both trying to run different migration schemas. Both convinced they were right.

And in honour of this moment, This meme was born :

// Detect dark theme var iframe = document.getElementById('tweet-1951981237419258332-512'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1951981237419258332&theme=dark" }

What We Should Have Done

1. Proper Load Balancing Architecture

Instead of our ad-hoc "some users go here, some go there" approach, we should have implemented:

# Ideal setup with external load balancer
External DNS (Cloudflare/Route53)
         |
    Load Balancer
    /           \
K8s Cluster    PB Server
    \           /
   Shared Database

This would have:

Automatically distributed traffic between both deployments
Provided seamless failover when K8s went down
Prevented the full outage scenario

2. Test Your Autoscaling (Or Don't Use It)

Let's be honest - we enabled autoscaling because it sounded cool. Did we need it? Probably not. Did we test it? Definitely not.

Lessons learned:

Autoscaling is not a silver bullet
Test scaling behaviors in staging first
Sometimes, vertical scaling > horizontal scaling for stateful apps

Post-Mortem Action Items

[ ] Apologise to Cyber Team
[ ] Gaslight the participants that it wasn't a server issue, it was their internet that was flaky
[ ] Remind myself to always use the latest version everywhere
[ ] Beat up Govind for not telling me about the custom migration script he ran last night secretly until the autoscale pods crashed.

Past the memes

PBCTF 4.0 taught us that running a CTF competition is as much about infrastructure resilience as it is about challenge quality. While our participants experienced 35 minutes of downtime, I think it was their internet at fault afterall.

To all future CTF organizers: Learn from our mistakes. Pin your versions. Test your scaling. And maybe, just maybe, don't autoscale unless you really need to.

Kubernetes on the cloud vs on bare metal : Deception 101

Akash Singh — Thu, 18 Sep 2025 07:41:10 +0000

The Great LoadBalancer Debate of 2:13 AM

It started innocently enough. My friend Govind and I were debugging our Kubernetes setup when he dropped this bombshell:

"Bro, my LoadBalancer is provisioned by Digital Ocean. It's not a K8s LoadBalancer service."

I stared at my screen. It was 2:13 AM. My brain was running on pure chai.

"Are you sure about what you're saying?" I typed back. "Why do you have LoadBalancer + Ingress then? What does it LoadBalance to?"

What followed was a 20-minute debate that would fundamentally change how I understood Kubernetes. We were both right. We were both wrong. And we were both about to discover that "Cloud Native" is the tech industry's greatest marketing scam.

# What Govind showed me
$ kubectl get svc -n ingress-nginx
NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP
ingress-nginx-controller  LoadBalancer   10.109.16.98   152.42.156.235

# My claim: "This is a K8s service"
# His claim: "This is a Digital Ocean Load Balancer"
# The truth: We were having different conversations

The debate got so heated that at 2:20 AM, Govind literally said, "Ye debate settle karna padega. Coming to your house."

At 2:21 AM. On a Sunday.

// Detect dark theme var iframe = document.getElementById('tweet-1959519918426067151-372'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1959519918426067151&theme=dark" }

Introduction

I am Akash Singh, a final year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

The Revelation That Changed Everything

After our debate (which we settled by asking Claude at 2:30 AM like real engineers), I decided to test something. I spun up the exact same Kubernetes cluster on a bare metal server.

Same YAML. Same configurations. Same everything.

# On Digital Ocean - Govind's setup
$ kubectl apply -f ingress-service.yaml
$ kubectl get svc
NAME                      TYPE           EXTERNAL-IP
ingress-nginx-controller  LoadBalancer   152.42.156.235  ✅

# On my bare metal server - The shocking truth
$ kubectl apply -f ingress-service.yaml  # EXACT SAME YAML
$ kubectl get svc
NAME                      TYPE           EXTERNAL-IP
ingress-nginx-controller  LoadBalancer   <pending>  ❌

One hour passed. Still <pending>.

That's when it hit me like a ton of YAML files: I was right. The LoadBalancer was a K8s service. But he was also right – it was provisioning a Digital Ocean Load Balancer.

The real mindfuck? On bare metal, it does... nothing. It just sits there. Pending. Forever.

type: LoadBalancer doesn't create a load balancer. It just asks your cloud provider to create one. No cloud provider? No load balancer. Just eternal pending and sadness.

The Architecture of Deception : Kubernetes on the cloud vs on bare metal

Here's what nobody tells you about type: LoadBalancer in Kubernetes: It doesn't create a load balancer. It just asks someone else to create one for you.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer  # <- This is just a prayer to the cloud gods
  ports:
    - port: 80

On AWS, this YAML whispers to the AWS Cloud Controller Manager, which calls the AWS API, which provisions an Elastic Load Balancer, which costs you $20/month, which... works.

On bare metal, this YAML whispers into the void. Nobody's listening. Nobody's home.

How Kubernetes Actually Works on Cloud

When you deploy Kubernetes on AWS, GCP, or Azure, there's a hidden component doing all the heavy lifting: the Cloud Controller Manager. It's like having a rich uncle who secretly pays for everything while you pretend to be independent.

Your YAML → K8s API → Cloud Controller Manager → AWS API → Real Infrastructure
                                ↑
                    This guy has your credit card

Every cloud resource you think Kubernetes is managing? It's not. It's just asking the cloud provider to do it:

LoadBalancer Service → Creates AWS ELB/ALB/NLB
PersistentVolume → Creates EBS volumes
Cluster Autoscaler → Calls EC2 APIs
Ingress → Often creates another load balancer

Your monthly AWS bill is basically a list of things Kubernetes couldn't do by itself lmao.

How Kubernetes "Works" on Bare Metal

On bare metal, that same YAML becomes a wish list with no Santa:

Your YAML → K8s API → ??? → Nothing happens → <pending> → Sadness

To make it work, you need to install the Kubernetes Extended Universe™:

MetalLB: Pretends to be a cloud load balancer
Longhorn/OpenEBS: Pretends to be cloud storage
Cluster API: Pretends to be cloud compute
Cert-Manager: Because clouds gave you free SSL
External-DNS: Because clouds handled DNS
Some IPAM solution: Because clouds managed IPs

By the time you're done, you've basically rebuilt AWS in your data center, poorly.

The Real Cost Comparison

Cloud Kubernetes:

# Time to Production: 10 minutes
# Monthly Cost: $500-2000
# Mental Health: Intact
# What You're Actually Paying For:
- EKS/GKE/AKS Control Plane: $75/month
- Load Balancers: $20 each
- NAT Gateways: $45 each  
- Data Transfer: $0.09/GB (the silent killer)
- Persistent Volumes: $0.10/GB/month
- The privilege of not thinking about any of this

Bare Metal Kubernetes:

# Time to Production: 3 days to 3 weeks
# Monthly Cost: $100-500 (hardware/colocation)
# Mental Health: What mental health?
# Hidden Costs Nobody Mentions:
- Your time: 40-80 hours of setup
- Ongoing maintenance: 10-20 hours/month
- That 3 AM phone call when the node dies
- The contractor you'll hire: $150/hour
- Replacing the keyboard you broke in rage

The Uncomfortable Solutions

Option 1: Embrace the Lock-in

# The path of least resistance
$ kubectl apply -f application.yaml
$ aws eks update-kubeconfig --name my-cluster
$ kubectl get svc  # Everything has IPs!
$ # Go home at 5 PM

Just use EKS/GKE/AKS and accept that you're paying the "I want to sleep at night" tax. For most companies, the $500-2000/month is cheaper than the engineering time you'd spend fighting bare metal.

Option 2: The Bare Metal Warrior Path

# The path of pain
$ # Install Ubuntu
$ # Install Kubernetes
$ # Install MetalLB
$ # Configure IP pools
$ # Install Longhorn
$ # Debug networking for 6 hours
$ # Question life choices
$ # Finally get a LoadBalancer IP
$ # Realize you need to do this on 5 more nodes

This path makes sense if:

You have dedicated ops people who enjoy suffering
You're at a scale where cloud costs exceed engineer salaries
You have compliance requirements that forbid cloud
You're a masochist

Option 3: The Reasonable Middle Ground

Use k3s, k0s, or MicroK8s. These distributions come with batteries included:

# k3s: Actually reasonable
$ curl -sfL https://get.k3s.io | sh -
$ # Includes Traefik (LoadBalancer alternative)
$ # Includes local-path storage
$ # Actually works on bare metal

Or just admit defeat and use Docker Compose for anything under 10 services.

The Plot Twist: When Cloud Kubernetes Actually Makes Sense

Let me be clear: I'm not saying Kubernetes on cloud is bad. For many scenarios, it's the right choice:

When Cloud Wins:

You need to scale from 10 to 10,000 pods based on traffic
You have a team of 5 managing infrastructure for 500 developers
Your compliance requires "high availability" (good luck doing that on bare metal)
You value your weekends

When Bare Metal Wins:

You're running a consistent workload 24/7 (cloud is just renting at that point)
You have very specific hardware requirements (GPUs, high-frequency trading)
You're at Facebook/Google scale (they use bare metal internally)
You're learning and want to understand everything

My Honest Recommendations

After suffering through both approaches, here's my actual advice:

For Startups (< 10 services)

# Skip Kubernetes entirely
$ docker-compose up -d
$ # Use the saved money for product development
$ # Graduate to K8s when you actually need it

For Growing Companies (10-100 services)

# Just pay for managed Kubernetes
$ eksctl create cluster --name my-cluster
$ # Focus on your product, not infrastructure
$ # The cloud bill is cheaper than hiring more ops people

For Large Enterprises (100+ services)

# You can afford to do it right
$ # Hire a platform team
$ # Build on bare metal if it saves millions
$ # Or stay on cloud if velocity matters more

For Learning

# Use kind or minikube locally
$ kind create cluster
$ # Learn the concepts without the pain
$ # Deploy to cloud when you need production

Conclusion: Making Peace with Reality

Special thanks to Govind for the 2 AM debate that inspired this post. Yes, we're still friends. No, we still don't agree on who won. And yes, Naresh is still paying for that Digital Ocean LoadBalancer.

After all my rage against the cloud native machine, here's where I've landed:

Use cloud Kubernetes if:

You value velocity over cost
You have variable workloads
You want to focus on your product
You like sleeping

Use bare metal Kubernetes if:

You have consistent workloads
You have a dedicated ops team
You're at massive scale
You enjoy pain (this is valid)

Use something else if:

You have less than 10 services (seriously, just use Docker Compose)
You don't need the complexity
You're a solo developer
You value simplicity

The truth is, "Cloud Native" isn't a lie – it's just poorly named. It should be called "Cloud Optimized" or "Cloud Enhanced" or, more honestly, "Only Really Works Properly On Cloud But We Can't Say That Because It Sounds Bad."

And that's okay. Not everything needs to be portable. Not everything needs to run on your laptop. Sometimes, paying AWS to handle the complexity is the smartest business decision you can make.

Just don't pretend it's portable. Don't pretend there's no lock-in. And definitely don't try to run LoadBalancer services on bare metal at 3 AM. Trust me on that last one.

Argo, Flux, Kustomize, Helm and all the fancy stuff about GitOps

Akash Singh — Fri, 15 Aug 2025 13:09:04 +0000

Every now and then, I open the CNCF Landscape and think: Would most non-engineers be able to differentiate if these are names of Software Engineering tools or Pokémon?

// Detect dark theme var iframe = document.getElementById('tweet-1955321080958030160-880'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1955321080958030160&theme=dark" }

So when I casually mentioned "I'm doing GitOps now" in a team meeting, I was immediately bombarded with:

"Oh, are you using Argo?"
"ArgoCD or Argo Workflows?"
"Why not Flux?"
"Flux v1 or v2?"
"Are you using Helm?"
"Helm with Kustomize or just Helm?"
"What about Jsonnet?"
"Have you tried Kapitan?"
"Don't forget Argo Events!"
"And Argo Rollouts!"

I nodded along, pretending I knew what everyone was talking about, frantically taking notes that looked like:

Argo (CD? Workflows? Events? Rollouts? Is this one tool or four???)
Flux (there are versions???)
Helm (⛵ boat emoji because lmao)
Kustomize (customize but with K because...Kubernetes???)

Three months and countless conversations, building and breaking later, I finally understand what all these tools actually DO. So I'm writing this blog to save you from the same confusion I went through. And yes Falco is straight up legendary Pokémon :D

Introduction

I am Akash Singh, a final year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

What Even Is GitOps?

Before we dive into all these tools, let's get one thing straight: GitOps is just a fancy way of saying "Git is the boss of your infrastructure."

Instead of SSHing into servers or running kubectl commands from your laptop, everything goes through Git:

Want to deploy? Push to Git.
Want to rollback? Revert the Git commit.
Want to know what's running in production? Look at Git.

The magic happens through pull-based deployment: Instead of your CI/CD pushing changes TO your cluster, your cluster pulls changes FROM Git. (This also means your CI/CD never needs production credentials hehe)

Now let me introduce slowly all the fancy keywords being thrown around here.

The Configuration Tools: How to Structure Your Kubernetes YAML

Before you can do GitOps, you need to organize your Kubernetes configurations.

Plain YAML: The Starting Point

This is just raw Kubernetes manifests. Cute YAML files.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: app
        image: nginx:1.14.2

What they do: Nothing (Just like my DNS server that I wrote in rust T-T)

The problem they creates: You end up copying these files for each environment (dev, staging, prod) and manually editing values. Lots of circus.

How it ties to GitOps: These YAML files live in Git, and GitOps tools apply them to your cluster.

Helm: The Package Manager

Helm treats Kubernetes applications like packages. My javascript bros found their npm for Kubernetes.

# Template
replicas: {{ .Values.replicas }}

# Values file
replicas: 3

What it does:

Templates your YAML using Go templates
Packages applications into "charts"
Manages releases (install, upgrade, rollback)
Handles dependencies between apps

How it ties to GitOps: Your Helm charts and values files live in Git. GitOps tools like ArgoCD/Flux can render and deploy Helm charts directly.

Kustomize: The Patcher

Kustomize says "don't template, just patch." Start with plain YAML and overlay changes.

# Base YAML stays plain
# Then you add patches for each environment
patches:
  - target:
      kind: Deployment
      name: my-app
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 5

What it does:

Takes base YAML files and applies patches/overlays
No templating language, All is YAML
Built into kubectl (kubectl apply -k) (this is the selling point fr)

How it ties to GitOps: Your base configs and overlays live in Git. GitOps tools natively understand Kustomize directories.

FAST FAST REVISION

Plain YAML: No tooling, lots of duplication
Helm: Powerful templating + package management, but another language to learn
Kustomize: Stay close to vanilla YAML, just patch what's different

The GitOps Engines: The Deployment Orchestrators

These tools watch your Git repos and make sure your cluster matches what's in Git.

ArgoCD: The GitOps Powerhouse

ArgoCD is probably what most people think of when they hear "GitOps."

// Detect dark theme var iframe = document.getElementById('tweet-1949927509971390487-134'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1949927509971390487&theme=dark" }

What it does:

Watches your Git repositories
Compares desired state (Git) with actual state (cluster)
Syncs changes automatically or manually
Provides a web UI to visualize everything

What makes it different:

The UI: Beautiful web interface showing your entire application topology
Multi-cluster support: Manage multiple clusters from one ArgoCD instance
Sync waves and hooks: Complex deployment orchestration
Built-in RBAC: Enterprise-ready from day one

How it works with config tools:

Automatically detects if you're using Helm, Kustomize, or plain YAML
Can pass values to Helm or parameters to Kustomize
Supports custom plugins for other tools

Flux: The Cloud Native GitOps

Flux takes a more modular, Kubernetes-native approach.

What it does:

Same core idea: syncs Git with your cluster
But built as a set of small, focused controllers
No UI - everything is a Kubernetes resource

What makes it different:

Architecture: Not a monolithic app but a collection of controllers:
- Source Controller (fetches from Git)
- Kustomize Controller (builds and applies)
- Helm Controller (manages Helm releases)
- Notification Controller (alerts and webhooks)
Multi-tenancy first: Designed for platform teams serving multiple dev teams
Lighter footprint: Uses fewer resources than ArgoCD
GitOps for GitOps: Flux can manage itself

How it works with config tools:

Native Kustomize support through Kustomize Controller
Native Helm support through Helm Controller
Can even fetch Helm charts from OCI registries

ArgoCD vs Flux:

Aspect	ArgoCD	Flux
UI	Beautiful web UI	No UI (but you can add one)
Architecture	Monolithic application	Set of controllers
Resource usage	Heavier	Lighter
Learning curve	Easier (thanks to UI)	Steeper (all CLI/YAML)
Multi-tenancy	Supported	First-class citizen
Self-management	Can manage itself	Designed to manage itself
Community	Larger, more tutorials	Smaller but growing

Choose ArgoCD if: You want a UI, easier onboarding, enterprise features out of the box

Choose Flux if: You prefer everything as code, need strong multi-tenancy, building a platform

// Detect dark theme var iframe = document.getElementById('tweet-1945459214434144285-798'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1945459214434144285&theme=dark" }

The Argo Ecosystem: Where the confusion happens

As it turns out, ArgoCD is actually part of a larger family. And as far as I have seen on the internet, People pick some members of this family to work with flux.

Argo Workflows: The Pipeline Runner

What it does: Runs complex, multi-step workflows on Kubernetes

Think of it as Jenkins or GitHub Actions, but Kubernetes-native. Each step in your workflow runs as a container.

What makes it different:

Workflows are Kubernetes resources (YAML)
DAG-based (define dependencies between steps)
Massive parallelization capabilities
Native Kubernetes integration (uses pods, volumes, configs)

Common use cases:

CI/CD pipelines
Data processing pipelines
Machine learning workflows
Batch jobs

How it ties to GitOps: Often triggered by Argo Events to build and push images, which then triggers ArgoCD to deploy them.

Argo Events: The Event Handler

What it does: Listens for events and triggers actions in Kubernetes

It's like IFTTT or Zapier for Kubernetes: "When this happens, do that."

What makes it different:

Huge variety of event sources:
- Webhooks (GitHub, GitLab, Slack)
- Message queues (Kafka, NATS, SQS)
- Time-based (cron, calendar)
- Cloud events (S3 uploads, pub/sub)
Can trigger any Kubernetes resource (including Argo Workflows)

Common use cases:

Trigger builds on Git push
Process files when uploaded to S3
Run nightly batch jobs
Incident response automation

How it ties to GitOps: Provides the event-driven glue between your source code changes and your GitOps deployments.

Argo Rollouts: The Progressive Delivery Tool

What it does: Advanced deployment strategies (canary, blue-green, experiments)

While ArgoCD gets your app deployed, Rollouts controls HOW it gets deployed.

What makes it different:

Gradual rollouts with automatic rollback
Integration with metrics providers (Prometheus, Datadog)
Traffic management with service meshes
A/B testing and experimentation

How it ties to GitOps: Works alongside ArgoCD/Flux to provide safe, gradual deployments.

How It All Fits Together

Here's how these tools work together in a real GitOps pipeline:

The Complete Flow

1. Developer pushes code to GitHub
2. Argo Events detects the push
3. Argo Events triggers Argo Workflows
4. Argo Workflows:
   - Builds the application
   - Runs tests
   - Builds and pushes Docker image
   - Updates image tag in the config repo
5. ArgoCD/Flux detects the config change
6. ArgoCD/Flux applies the changes to Kubernetes
7. Optionally: Argo Rollouts performs canary deployment

Repository Structure in GitOps

You typically have two repos:

Application Repository : Owned by Developers

my-app/
├── src/
├── Dockerfile
└── .github/workflows/  # or Argo Workflows

Configuration Repository : Owned by the cool kids (Devops Team)

k8s-configs/
├── apps/
│   └── my-app/
│       ├── base/           # Kustomize base
│       │   └── deployment.yaml
│       ├── overlays/       # Kustomize overlays
│       │   ├── dev/
│       │   └── prod/
│       └── charts/         # Or Helm charts
└── argocd/                 # ArgoCD applications
    └── apps/

Tool Combinations in the Wild

Common Pattern 1: The Argo Stack

ArgoCD for GitOps
Argo Workflows for CI/CD
Argo Events for automation
Kustomize for configuration

Common Pattern 2: The Flux Stack

Flux for GitOps
GitHub Actions/GitLab CI for CI/CD
Helm for third-party apps
Kustomize for internal apps

Common Pattern 3: The Hybrid

Flux for platform/infrastructure (used by platform team)
ArgoCD for applications (used by dev teams - they get a UI!)
Helm for third-party apps
Kustomize for customization

So Which Pokemon do I choose?

You don't need all these tools. Start with ArgoCD/Flux + Kustomize. That's enough for 80% of use cases.
The tools are the easy part. The hard part is changing your team's workflows and getting everyone to stop running kubectl manually.
Secret management is still painful. You'll need something like Sealed Secrets, SOPS, or External Secrets Operator.
Image tag updates are annoying. Every new build means a commit to your config repo. (Both Flux and ArgoCD have solutions for this.)

Don't try to build the perfect GitOps platform on day one. You will shoot yourself in the foot.

But if you are like me and still want to break things just for the sake of it :

The TL;DR of the mess

GitOps: Git is your source of truth, clusters pull from Git
Helm vs Kustomize: Helm for packages and templating, Kustomize for patching
ArgoCD vs Flux: ArgoCD for nice UI and easy start, Flux for platform building
Argo Family: Workflows for pipelines, Events for automation, Rollouts for progressive delivery

Remember: These tools exist to solve problems. If you don't have the problem, you don't need the tool. Start with your pain points and add tools as needed. As engineers, it is our no. 1 goal to solve actual problems instead of creating 20 new problems with an over-engineered tech-stack.

DreamOps: The AI Agent That Fixes the Oncall Circus

Akash Singh — Wed, 25 Jun 2025 13:49:42 +0000

The Circus of Being Oncall

Picture this: It's 3 AM. Your phone buzzes with that dreaded PagerDuty alert. Your production database is down, users are angry, and you're stumbling in the dark trying to diagnose what went wrong. Sound familiar?

This is the reality for thousands of on-call engineers worldwide:

Constant sleep interruptions and alert fatigue
Manual log analysis across multiple systems under pressure
30-60 minutes of stressful debugging for common issues
Inconsistent remediation quality when you're exhausted
Burnout from repetitive tasks that could be automated

We built DreamOps to solve this exact problem. And the results? Mind-blowing.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

Meet DreamOps: Your AI-Powered On-Call Partner

DreamOps is an intelligent incident response platform that automatically triages and resolves infrastructure issues using Claude AI and advanced integrations. Think of it as having a senior DevOps engineer who never sleeps, never gets tired, and learns from every incident.

🎯 The Impact

80% faster incident resolution (2-5 minutes vs 30-60 minutes)
2-4 hours saved per on-call shift
Zero 3 AM wake-up calls for routine issues
Consistent remediation quality regardless of time of day

// Detect dark theme var iframe = document.getElementById('tweet-1936766682464428497-731'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1936766682464428497&theme=dark" }

🔧 How It Works

When PagerDuty sends an alert, our AI agent:

Instantly analyzes the incident with full Kubernetes context
Diagnoses root cause using logs, metrics, and documentation
Executes remediation commands automatically (with safety checks)
Only escalates truly complex issues that need human intervention

The Tech Behind the Magic ✨

AI-First Architecture

Claude AI Integration: Advanced reasoning for root cause analysis
Model Context Protocol (MCP): Seamless integration with 10+ tools
Confidence Scoring: Only auto-executes actions with ≥80% confidence
Risk Assessment: Categorizes commands as low/medium/high risk

Production-Ready Stack

Backend: Python FastAPI with async processing
Frontend: Next.js SaaS interface with real-time dashboards
Infrastructure: AWS ECS/EKS deployment ready
Integrations: Kubernetes, PagerDuty, Grafana, GitHub, Slack, Notion

YOLO Mode 🎢

Yes, we actually called it YOLO mode. When enabled, DreamOps autonomously executes remediation commands for common issues like:

Pod crashes (CrashLoopBackOff)
Memory issues (OOMKilled)
Configuration problems
Deployment failures

But don't worry - it's safer than it sounds. Every action is risk-assessed and confidence-scored.

From Hackathon Glory to Production Reality

The Lightspeed Warpseed 2025 Victory 🏆

This project didn't just emerge from our shared frustration with traditional incident response - it was born in the crucible of competition. At the Lightspeed Warpseed 2025 hackathon, we took our 3 AM debugging nightmares and turned them into a winning solution.

The result? We won $3,000 USD and validation that we'd struck gold.

The hackathon judges were blown away by our approach to solving a problem that every engineer in the room had experienced. While other teams built incremental improvements, we reimagined incident response from the ground up with AI at the core.

The Hackathon Journey

We've all been there - debugging production issues at ungodly hours, making critical decisions while sleep-deprived. During the hackathon, we:

Identified the core pain point that affects millions of engineers worldwide
Leveraged cutting-edge AI (Claude) in ways no one had attempted before
Built a working prototype that actually resolved real Kubernetes issues
Demonstrated measurable impact with our 80% faster resolution times

The hackathon victory wasn't just about the prize money - it was proof that the developer community desperately needed this solution.

From Prototype to Platform

What started as a 48-hour hackathon sprint has evolved into a comprehensive platform that's changing how teams handle incidents. The $3,000 prize was just the beginning - we've since invested every dollar back into making DreamOps production-ready.

🔗 Check out our journey:

Project Repository (Currently private - building in stealth mode)
Pitch Presentation
Demo Video
Devfolio Project

// Detect dark theme var iframe = document.getElementById('tweet-1937087277215416359-404'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1937087277215416359&theme=dark" }

Real-World Results That Speak Volumes

Before DreamOps:

45-minute average incident resolution time
Engineers woken up 3-5 times per night
Inconsistent fixes due to human error under pressure
High on-call stress and burnout rates

After DreamOps:

5-minute average resolution for common issues
90% reduction in middle-of-night escalations
Standardized, tested remediation procedures
Engineers actually getting sleep 😴

"DreamOps doesn't just solve incidents faster - it learns from each one to prevent future occurrences. It's like having a senior engineer who gets smarter with every alert." - The Team

What's Next: Building the Future of Incident Response

We're not stopping here. The hackathon victory was just the beginning - DreamOps is evolving into the definitive platform for intelligent infrastructure management.

Post-Hackathon Roadmap:

🔮 Predictive Incident Prevention: Stop issues before they happen
🌐 Multi-Cloud Support: AWS, GCP, Azure integration
📊 Advanced Analytics: Cost impact analysis and SLO tracking
🤝 Team Collaboration: Intelligent escalation and knowledge sharing
🛡️ Security Integration: Automated security incident response

Looking for Strategic Partners & Investors

Our hackathon victory proved market demand - now we're scaling.

We're actively seeking investors and strategic partners who understand the massive pain point we're solving. The incident response market is ripe for disruption, and early adopters are seeing transformational results.

Why invest in DreamOps?

🏆 Proven concept: $3,000 hackathon winner with judge validation
📈 Massive market: $2B+ incident management market growing 15% annually
🎯 Demonstrated traction: Real results from early adopters
🚀 AI-first approach: Leveraging the latest advances in LLMs
👥 Experienced team: Deep DevOps and AI expertise
🔧 Production-ready: Not just a prototype - full enterprise platform

Experience DreamOps Today

Ready to revolutionize your incident response? Here's how to get started:

For Teams:

Quick Setup: Deploy in under 30 minutes
Pilot Program: Start with non-critical alerts
Gradual Rollout: Expand to full production workloads
Sleep Better: Enjoy uninterrupted nights

For Investors:

Schedule a demo call with our team
Review our pitch deck and financials
Meet our early adopters and hear their stories
Join us in transforming how the world handles incidents

The Team Behind the Magic

Sky Singh - Lead Developer

Inchara J - AI/ML Engineer

Himanshu - Frontend Developer

Harsh Kumar Gupta - Backend Systems

Shubhang Sinha - Cancelled on us

A diverse team united by a shared mission: making on-call duty humane again. Our hackathon victory proved we have the skills - now we're building the future.

Get Involved

Whether you're an engineer tired of 3 AM alerts, a CTO looking to improve team productivity, or an investor seeking the next big DevOps breakthrough - we want to connect.

From hackathon winners to your production environment - let's build the future of incident response together.

📧 Contact us: [Insert contact information]

🐦 Follow our journey: @SkySingh04

💼 Investment inquiries: [Insert investor contact]

🔧 Early access: [Insert beta signup link]

The future of incident response is here. It's intelligent, it's automated, and it lets you sleep through the night.

Ready to dream easy while AI takes care of your on-call duty?

DreamOps - Because 3 AM debugging sessions should be a thing of the past. ✨

P.S. - We're still celebrating our Lightspeed Warpseed 2025 victory, but we're more excited about the problems we're solving for engineers worldwide. Join us on this journey!

// Detect dark theme var iframe = document.getElementById('tweet-1936766175125631017-649'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1936766175125631017&theme=dark" }

The Art Of Chaos Engineering : LFX 2025 Experience

Akash Singh — Sun, 01 Jun 2025 11:27:09 +0000

I was selected for the LFX Mentorship Program 2025 under LitmusChaos where my project was CNCF - LitmusChaos: CI/CD Integration, SDK Development & Chaos-CI-Lib Revamp

During my mentorship, I submitted a total of 7 PR's to 4 repositories of LitmusChaos with approximately 14k lines of code. A detailed gist containing all of my contribution PR's can be found here.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

Application Process

The LFX Mentorship application journey began in December 2024 when applications opened for the 2025 Term 1 (March-May) cohort. The process was both exciting and nerve-wracking as I navigated through multiple project applications.

Projects I Applied To

Having prior open-source contributions gave me the confidence to apply to three CNCF projects:

LitmusChaos: CI/CD Integration, SDK Development & Chaos-CI-Lib Revamp - This project aimed to revolutionize the CI/CD experience for chaos engineering by developing a dedicated SDK and modernizing the Chaos-CI-Lib for Litmus 3.x compatibility.
KCL: KPM & LSP Integrated - Focused on strengthening the integration between Language Server Protocol (LSP) and KCL Package Manager (KPM) for better IDE experience.
Karmada: Self-Signed Certificate Content Standardization - Aimed at enhancing Karmada's certificate system compliance by ensuring each component has distinct certificates.

Application Requirements

The LFX portal required two components for the project's I appiled to:

Resume: Highlighting relevant technical skills and open-source contributions
Cover Letter: Explaining my motivation and how my skills aligned with each project

Note that these vary from org to org. Many orgs also conduct an interview round / give additional onboarding tasks that you have to complete.

The Selection

After weeks of anticipation, I received the selection email in late February 2025! The mentors - Shubham Chaudhary and [Vedant https://github.com/Jonsy13) from LitmusChaos - had evaluated all applications and selected me for the chaos engineering project.

What made my application stand out was:

My existing contributions to Karmada and KCL, demonstrating familiarity with CNCF projects
Strong Go programming skills essential for SDK development
Understanding of Kubernetes and CI/CD pipelines
Clear communication in my cover letter about how I could contribute to the project's goals

What is Chaos Engineering?

This mentorship was my first deep dive into Chaos Engineering and honestly I find it such a cool concept in Software Engineering and Testing in general. The way I explain it to my juniors at Point Blank is its' the process of experimenting on a distributed system to build confidence in the system's capability to withstand ✨circus✨ conditions in production. It's about intentionally injecting failures into your systems to discover weaknesses before they impact your customers and you get a call at 3am.

It is as simple as a fire drill for your infrastructure - you don't wait for an actual fire to test your emergency procedures. Similarly, chaos engineering helps you proactively identify and fix potential issues (what I refer to as ✨circus✨ conditions).

Me teaching my juniors at Point Blank about Chaos Engineering
// Detect dark theme var iframe = document.getElementById('tweet-1928079304451313876-778'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1928079304451313876&theme=dark" }

Why LitmusChaos?

LitmusChaos is a CNCF incubating project that makes chaos engineering easy for developers and SREs. What makes it special:

Cloud-Native: Built specifically for Kubernetes environments
Extensible: Large library of pre-built experiments
GitOps Friendly: Declarative chaos workflows
Enterprise Ready: RBAC, audit logs, and multi-tenancy support

When I first encountered LitmusChaos, I was fascinated by how it democratizes chaos engineering, making it accessible to teams of all sizes.

My Project: Modernizing LitmusChaos from 1.0 to 3.0 Approach

My mentorship project focused on three critical areas:

1. SDK Development

I built a comprehensive Go SDK from scratch that provides a programmatic interface to LitmusChaos. This SDK enables developers to:

Authenticate and manage projects
Create and execute chaos experiments
Manage infrastructure and environments
Monitor experiment status in real-time

2. CI/CD Integration

I revamped the Chaos-CI-Lib to align with Litmus 3.0, transitioning from direct Kubernetes API calls to a cleaner SDK-based approach. This included:

Refactoring the entire codebase from 1.0 approach to 3.0 approach that uses our SDK.
Removing direct invocations of K8s API Calls with our SDK approach.
Running and testing over 13 experiments end to end with the refactoring.

3. Template Optimization

Updated GitHub Actions and GitLab CI templates to work seamlessly with Litmus 3.0, adding:

Enhanced configuration options
Better infrastructure management
Comprehensive probe configurations

My Experience: From Zero to Chaos Engineer

The Learning Curve

Coming into this project, I had experience with Go and Kubernetes, but chaos engineering was new to me. The first few weeks were intense - understanding GraphQL APIs, diving deep into Kubernetes operators, and grasping the architecture of a distributed chaos engineering platform.

Key Achievements

Built a production-ready SDK: The litmus-go-sdk now serves as the foundation for all programmatic interactions with LitmusChaos
Refactored and Rewrote Chaos-Ci-Lib: Rewrote and released the new version of chaos-ci-lib with our Litmus 3.0 approach using our fancy new SDK.
Modernized CI/CD workflows: Reduced setup complexity for users integrating chaos testing into their pipelines on both Github Actions and Gitlab Pipelines.

Gratitude

This incredible journey wouldn't have been possible without:

Shubham Chaudhary (@ispeakc0de) - For his patient guidance and code reviews that taught me industry best practices
Vedant Shrotria (@Jonsy13) - For helping me understand the deeper architecture of LitmusChaos
Sayan Mondal (@S-ayanide) - For coordinating the mentorship program and ensuring smooth progress

Special thanks to the entire LitmusChaos community for being welcoming and supportive throughout my journey.

Conclusion

The LFX mentorship program has been the most rewarding experience of my year. It has transformed me from a beginner in chaos engineering to a confident contributor capable of making meaningful improvements. This experience enhanced my technical understanding of chaos engineering and helped me get insights into my code and infrastructure. I have learned that anything is achievable with the right amount of energy and time.

My journey with LitmusChaos doesn't end here - I'm committed to continuing my contributions to the project. I plan to maintain the SDK, help onboard new contributors, work on new features, and contribute to the ecosystem's growth. The mentorship might be ending, but my involvement with the LitmusChaos community is just beginning!

Don't hesitate to apply for this mentorship for the next quarter - it could be the start of your own incredible open-source journey!

How I reduced $10000 Monthly AWS Glue Bill to $400 using Airflow

Akash Singh — Sat, 15 Feb 2025 12:38:50 +0000

During my time as a Devops Engineer at Vance, we were running around 80 ETL pipelines on AWS Glue, but as our workloads scaled, so did our costs—hitting a staggering $10,000 per month. This wasn’t sustainable. After analyzing our pipeline, we realized that AWS Glue's serverless nature was costing us heavily for idle time and unnecessary compute

To fix this, I migrated our ETL workloads to Apache Airflow, running on EC2 instances with ECS, and orchestrated everything using Terraform. The result? A 96% cost reduction, bringing our bill down to just $400 per month, without compromising on performance.

While Airflow is a great alternative to Glue, there's little documentation on setting it up properly with Terraform with Celery Executor — especially for cost optimization. This blog walks you through how we did it, the challenges we faced, and how you can do the same to slash your AWS Glue costs.

Needless to say, this was indeed a war, credits to my manager Rishabh Lakhotia who went through this circus with me, you are indeed a god sir.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

The three parts of the pain

Migrating from AWS Glue to Apache Airflow involves setting up three core components:

Webserver – The UI for managing DAGs (Directed Acyclic Graphs) and monitoring job execution.
Scheduler – Responsible for triggering and scheduling DAG runs.
Workers – Execute the actual tasks in the DAGs.

Using Terraform, we provisioned ECS to run all three parallely and enable them to communicate with each other, which we will get to next.

Once Airflow is up and running, the next step is to migrate our ETL workflows. We will Glue jobs into Airflow DAGs, and then nuke the the Glue jobs, marking the final step in cutting down our AWS costs.

The Magical Dockerfile

You can use the following Dockerfile and push it to ECR and reference it in the upcoming configs :

FROM apache/airflow:latest-python3.9
USER root
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    git \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /opt/airflow/dags /opt/airflow/logs && \
    chown -R airflow:root /opt/airflow && \
    chmod -R 755 /opt/airflow/logs


USER airflow

RUN pip install --no-cache-dir \
    apache-airflow-providers-github \
    apache-airflow-providers-amazon \
    apache-airflow-providers-mysql \
    apache-airflow-providers-mongo \
    apache-airflow[celery,redis] \
    pandas

COPY --chown=airflow:root dags/* /opt/airflow/dags/

ENV AIRFLOW__LOGGING__BASE_LOG_FOLDER=/opt/airflow/logs \
    AIRFLOW__LOGGING__WORKER_LOG_SERVER_PORT=8793 \
    AIRFLOW__LOGGING__LOGGING_LEVEL=INFO \
    AIRFLOW__LOGGING__LOG_FORMAT='[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s' \
    AIRFLOW__LOGGING__SIMPLE_LOG_FORMAT='%(asctime)s %(levelname)s - %(message)s' \
    AIRFLOW__LOGGING__DAG_PROCESSOR_LOG_TARGET=file \
    AIRFLOW__LOGGING__TASK_LOG_READER=task \
    AIRFLOW__LOGGING__DAG_FILE_PROCESSOR_LOG_TARGET=/opt/airflow/logs/dag_processor_manager/dag_processor_manager.log \
    AIRFLOW__LOGGING__DAG_PROCESSOR_MANAGER_LOG_LOCATION=/opt/airflow/logs/dag_processor_manager/dag_processor_manager.log

ENV AIRFLOW__CORE__DAGS_FOLDER=/opt/airflow/dags

RUN mkdir -p /opt/airflow/logs/scheduler \
            /opt/airflow/logs/web \
            /opt/airflow/logs/worker \
            /opt/airflow/logs/dag_processor_manager \
            /opt/airflow/logs/task_logs


USER root
RUN chown -R airflow:root /opt/airflow && \
    chmod -R 755 /opt/airflow
USER airflow

This Dockerfile will be used for all three of our components and very nicely setups logging as well. The DAGs are directly baked into the Docker Image, we will get to that in a bit. Builld the image, tag it, push it to ECR and move to the next step!

// Detect dark theme var iframe = document.getElementById('tweet-1880591497571246356-981'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1880591497571246356&theme=dark" }

Airflow Web Server

A Terraform script can be written to set up Apache Airflow on AWS using ECS (Elastic Container Service) with EC2 launch type. We need to make sure to add:

CloudWatch Logging:
- Creates a log group (/ecs/airflow) with a retention of 3 days.
Security Groups:
- Allows inbound HTTP (port 80) and HTTPS (port 443) traffic for the Application Load Balancer (ALB).
- Enables unrestricted outbound traffic.
TLS/SSL with ACM & Route 53:
- Provisions an ACM (AWS Certificate Manager) certificate for airflow.internal.example.com using DNS validation.
- Configures Route 53 DNS records to resolve the Airflow URL to the ALB.
Application Load Balancer (ALB):
- Creates an internal ALB for the Airflow webserver, supporting IPv4 & IPv6 (dualstack).
- Configures an HTTP listener (port 80) to redirect traffic to HTTPS (port 443).
- Sets up an HTTPS listener (port 443) to forward requests to the ECS target group.
ECS Task Definition for Airflow Webserver:
- Defines an ECS task for the Airflow webserver running on an EC2-backed ECS cluster.
- Uses a Docker image stored in AWS ECR (aws_ecr_repository.airflow.repository_url:latest).
- Allocates 2GB memory (2048MB).
- Maps container port 8080 to the host for web access.
- Defines a health check (http://localhost:8080/health).
ECS Service for Airflow:
- Creates an ECS service named "airflow-webserver" with 1 desired task.
- Associates the ECS service with the ALB target group for load balancing.
- Enables execute-command to allow debugging via AWS SSM.
- Uses a capacity provider strategy for ECS resource management.
DNS Configuration:
- Configures a Route 53 A record (airflow.internal.example.com) pointing to the ALB.

The Terraform script includes several environment variables in the ECS task definition:

Database Connection (AIRFLOW__DATABASE__SQL_ALCHEMY_CONN):
- Specifies the PostgreSQL database connection string for Airflow’s metadata database.
- Uses AWS KMS-encrypted secrets to securely store the database password.
User Management:
- _AIRFLOW_WWW_USER_CREATE: Ensures the default Airflow web user is created.
- _AIRFLOW_WWW_USER_USERNAME: Sets the username (default: airflow).
- _AIRFLOW_WWW_USER_PASSWORD: Stores the password securely via AWS KMS secrets.
Security & Web Configuration:
- AIRFLOW__WEBSERVER__EXPOSE_CONFIG: Enables exposing Airflow configuration via the web UI.
- AIRFLOW__SCHEDULER__ENABLE_HEALTH_CHECK: Enables a built-in scheduler health check.
Database Migrations & Initialization:
- _AIRFLOW_DB_MIGRATE: Ensures Airflow runs necessary database migrations on startup.

Now go ahead and run terraform plan and terraform apply and you should see a lot of resources being created. If everything went correctly, you will see the airflow ui on the url you specified :

Airflow Scheduler

The Airflow Scheduler is responsible for orchestrating DAG executions and ensuring scheduled tasks run at the correct time. A Terraform script can be written to provision the scheduler as an ECS service, configures CloudWatch logging, and enables auto-scaling to manage resource usage effectively.

While most of this is similar to the webserver, in summary, we need to added :

Logs Scheduler Execution in CloudWatch (/ecs/airflow-scheduler/).
Monitors Performance via StatsD Metrics (airflow-metrics namespace).
Runs in an ECS Cluster with Auto Scaling, ensuring efficient resource allocation.
Uses CloudWatch Agent for Monitoring, helping analyze task execution times.
Secured by a Restricted Security Group, blocking unwanted traffic.

Now, go ahead and run terraform plan and terraform apply, and the Airflow Scheduler will be provisioned successfully! 🚀

Airflow Worker

The Airflow worker service is deployed as an ECS service on EC2 instances with auto-scaling based on memory utilization. It runs the Celery workers, which execute tasks from the DAGs, and will require Redis, which we’ll set up next.

The important things to note are :

Uses CeleryExecutor, meaning tasks are distributed among workers.
Logs are sent to CloudWatch for monitoring.
Workers scale dynamically between 0 and 5 based on memory utilization.
Each worker runs as a container inside ECS, managed by an autoscaling policy with a target of 60% memory utilization.
DUMB_INIT_SETSID=0 is set to handle proper signal propagation for Celery shutdown.

This entire setup made me cry because debugging autoscaling, log management, and task execution in ECS is a nightmare. Also, Redis isn’t even here yet, so the pain is far from over.

Redis and RDS

Setting up redis isn't that bad , you can use the following terraform file:

resource "aws_elasticache_subnet_group" "airflow" {
  name = "airflow-redis-subnet-group"
  subnet_ids = aws_subnet.airflow[*].id
  tags = merge(
    {
      name = "airflow-redis-subnet-group"
    },
    local.common_tags
  )
}

resource "aws_security_group" "airflow_redis" {
  name_prefix = "airflow-redis"
  vpc_id      = data.aws_vpc.this.id

  tags = merge(
    {
      Name = "airflow-redis"
    },
    local.common_tags
  )
}

resource "aws_security_group_rule" "airflow_redis_inbound" {
  type              = "ingress"
  from_port         = 6379
  to_port           = 6379
  protocol          = "tcp"
  cidr_blocks       = [data.aws_vpc.this.cidr_block]
  security_group_id = aws_security_group.airflow_redis.id
  description       = "Allow Redis from internal network"
}

resource "aws_elasticache_cluster" "airflow" {
  cluster_id           = "airflow"
  engine               = "redis"
  node_type            = "cache.t4g.small"
  num_cache_nodes      = 1
  parameter_group_name = "default.redis5.0"
  engine_version       = "5.0.6"
  port                 = 6379
  subnet_group_name    = aws_elasticache_subnet_group.airflow.name
  security_group_ids = [aws_security_group.airflow_redis.id]
  tags = merge(
    {
      name = "airflow-redis-server"
    },
    local.common_tags
  )
}

resource "aws_security_group_rule" "airflow_redis_outbound" {
  type              = "egress"
  from_port         = 0
  to_port           = 0
  protocol          = "-1"
  cidr_blocks       = ["0.0.0.0/0"]
  security_group_id = aws_security_group.airflow_redis.id
}

And similarly, we will setup RDS as well for airflow:

# Security Groups
resource "aws_security_group" "airflow_rds" {
  lifecycle {
    create_before_destroy = true
  }
  name_prefix = "airflow-rds-default-"
  description = "Allow TLS inbound traffic and all outbound traffic for airflow"
  vpc_id      = data.aws_vpc.this.id

  tags = {
    Name = "airflow-rds-default"
  }
}

resource "aws_security_group_rule" "airflow_rds_inbound" {
  type              = "ingress"
  from_port         = 0
  to_port           = 0
  protocol          = "-1"
  cidr_blocks       = [data.aws_vpc.this.cidr_block]
  security_group_id = aws_security_group.airflow_rds.id
  description       = "Allow all from internal network"
}

resource "aws_db_subnet_group" "airflow" {
  name       = "postgres-airflow"
  subnet_ids = aws_subnet.airflow[*].id
}


resource "aws_db_instance" "airflow" {
  db_name                    = "any db name"
  apply_immediately = true
  allocated_storage          = "100"
  storage_type               = "gp3"
  engine                     = "postgres"
  engine_version             = "17.2"
  auto_minor_version_upgrade = true
  instance_class             = "db.t4g.micro"
  username                   = "airflow"
  password                   = data.aws_kms_secrets.airflow.plaintext["db_password"]
  multi_az                   = false
  publicly_accessible        = false
  deletion_protection        = false
  skip_final_snapshot        = true
  identifier                 = "airflow"
  vpc_security_group_ids     = [aws_security_group.airflow_rds.id]
  db_subnet_group_name       = aws_db_subnet_group.airflow.name
}

Go ahead and create all of these resources as well using terraform!

The ENV Configurations to make all of this work

To make Airflow work properly in ECS with CeleryExecutor, several environment variables are required for logging, task execution, database connections, Redis as the message broker, and external integrations. These are defined in Terraform locals and passed into the Airflow containers.

1️⃣ Core Airflow Configuration

Instance Name:
- "AIRFLOW__WEBSERVER__INSTANCE_NAME" = "airflow-webserver"
- Helps identify the webserver instance.
Executor:
- "AIRFLOW__CORE__EXECUTOR" = "CeleryExecutor"
- Uses CeleryExecutor to distribute tasks across multiple workers instead of running them sequentially in a single instance.
Database Connection:
- "AIRFLOW__CORE__SQL_ALCHEMY_CONN"
- Connects to PostgreSQL, using credentials stored in AWS KMS secrets.
Load Examples:
- "AIRFLOW__CORE__LOAD_EXAMPLES" = "True"
- Controls whether example DAGs should be loaded.

2️⃣ Logging Configuration (AWS CloudWatch & S3)

Log Level:
- "AIRFLOW__LOGGING__LOGGING_LEVEL" = "DEBUG"
- Enables verbose logging for debugging.
Remote Logging to CloudWatch:
- "AIRFLOW__LOGGING__REMOTE_LOGGING" = "True"
- "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID" = "aws_conn"
- "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER" = "s3://abc"
- Stores logs in S3 and CloudWatch, making them accessible even if containers restart.

3️⃣ Celery & Redis Configuration (Message Queue & Task Result Storage)

Message Queue (Redis):
- "AIRFLOW__CELERY__BROKER_URL" = "redis://${aws_elasticache_cluster.airflow.cache_nodes[0].address}:6379/0"
- Celery uses Redis for task queuing (yet to be set up, another source of pain).
Task Result Storage (PostgreSQL):
- "AIRFLOW__CELERY__RESULT_BACKEND" = "db+postgresql://airflow:${data.aws_kms_secrets.airflow.plaintext["db_password"]}@${aws_db_instance.airflow.endpoint}/airflow"
- Task execution results are stored in PostgreSQL, ensuring persistence.
Celery Transport Options:
- "AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT" = "1800"
- Ensures tasks are not marked as failed too soon.

4️⃣ SMTP (Email Alerts for DAG Failures & Notifications)

SMTP Configuration:
- "AIRFLOW__SMTP__SMTP_HOST" = "d"
- "AIRFLOW__SMTP__SMTP_MAIL_FROM" = "abc@email.com"
- "AIRFLOW__SMTP__SMTP_PORT" = "587"
- "AIRFLOW__SMTP__SMTP_SSL" = "True"
- Used for sending failure notifications via email.

5️⃣ AWS-Specific Configurations

Region Setting:
- "AWS_DEFAULT_REGION" = local.region
- Ensures Terraform and Airflow components run in the correct AWS region.
Fluent Bit Logging for Observability:
- Uses Fluent Bit (aws-for-fluent-bit:stable) for log collection.

6️⃣ External Integrations (GitHub & AWS Secrets Manager)

GitHub Connection (Airflow Providers):
- "AIRFLOW__PROVIDERS__GITHUB__GITHUB_CONN_ID" = "github_default"
- "AIRFLOW__PROVIDERS__GITHUB__ACCESS_TOKEN" = data.aws_kms_secrets.airflow.plaintext["github_token"]
- Enables Airflow DAGs to interact with GitHub APIs.

Pain Points 😭

Setting up Redis for Celery is a huge pain because of networking and IAM role issues.
Debugging log storage in S3 & CloudWatch while handling permissions is frustrating.
Managing AWS Secrets Manager & KMS decryption for credentials adds complexity.
Auto-scaling workers based on Redis queue depth & CPU/memory usage needs fine-tuning.

Okay yes lets finally move the DAGs now!

Now that the Airflow infrastructure is mostly set up (minus the Redis pain 😭), it's time to move our DAGs. Instead of mounting them dynamically, we are baking them directly into the Docker image. This ensures that every container running the Airflow scheduler or worker has the DAGs preloaded without relying on external storage.

1️⃣ How We Bake DAGs into the Docker Image

In our Dockerfile (which we wrote earlier), we add the DAGs by copying them into the /dags directory inside the container

// Detect dark theme var iframe = document.getElementById('tweet-1886127124714766686-29'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1886127124714766686&theme=dark" }

2️⃣ Why This Approach?

✅ No need for external DAG storage (like S3, EFS, or Git sync).
✅ Ensures version control—DAGs are part of the Docker build process, so each deployment gets a known DAG version.
✅ Simplifies deployments—no extra steps to copy DAGs at runtime.

3️⃣ Building & Pushing the Docker Image

Once the DAGs are added, we build and push the image:

docker build -t airflow-custom:latest .
docker tag airflow-custom:latest <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/airflow:latest
docker push <AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.amazonaws.com/airflow:latest

4️⃣ Updating ECS to Use the New Image

Since we bake the DAGs into the image, we just update ECS to pull the latest image, and the DAGs will be there.

aws ecs update-service \
  --cluster airflow-cluster \
  --service airflow-scheduler \
  --force-new-deployment

This triggers a rolling restart of the scheduler, ensuring that the new DAGs are loaded.

Pain & Next Steps 😭

DAG Debugging: If a DAG has syntax errors, ECS will restart the scheduler on loop until it's fixed.
Hot Reloading? Baking DAGs means redeploying on every DAG update—fine for now, but we might add a mounted volume or Git sync later.
Testing DAGs Before Baking: To avoid bad deployments, we should test DAGs locally before adding them to the image.

Final Push: DAGs Are Moving In! 🏠

DAGs are now inside the container, meaning no runtime copying, no missing DAG issues, and one less thing to worry about—until the next fire starts. 🔥

// Detect dark theme var iframe = document.getElementById('tweet-1886761377832034352-972'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1886761377832034352&theme=dark" }

And Now I Rest on a Pile of Blood and Bodies ⚰️

It took two brutal days, but we slowly closed every Glue job, one by one, like a sniper picking off targets. Each shutdown was met with anticipation—would Airflow take over without issues, or would we be diving into yet another debugging nightmare?

With each transition, we watched the DAGs spin up, monitored task executions, and prayed that Celery wouldn't betray us. Logs were combed through, retries were tweaked, and countless cups of coffee were consumed.

And finally, at the end of this war, the numbers spoke for themselves:

🚀 96% cost reduction in Glue job expenses.

🔥 Airflow fully operational, with tasks running efficiently across ECS workers.

💀 Redis survived (barely), after nearly making us lose our minds.

This migration wasn’t just a deployment; it was a battle of wills, and somehow, against all odds, we came out victorious. Now, as the dust settles, I take a breath and rest—not because the war is over, but because the next battle is just around the corner.

The Magic behind Bzip2 : Burrows Wheeler Transform

Akash Singh — Mon, 30 Dec 2024 12:18:19 +0000

ACM Winter School 2024

I recently went to ACM Winter School 2024 and over there Prof. Chirag Jain showed us how Burrows Wheeler Transform works for compression in tools like bzip2 and how it can be used to optimize Suffix Trees and Suffix Arrays. In this blog, we'll unravel the workings of the BWT and its significance, while I try to translate my class notes into this blog xD.

// Detect dark theme var iframe = document.getElementById('tweet-1873089694227141066-138'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1873089694227141066&theme=dark" }

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

What is the Burrows-Wheeler Transform?

The BWT is a reversible text transformation that rearranges the characters of a string into runs of similar characters, making the string more amenable to compression. This transformation is pivotal in compression algorithms like bzip2 and indexing structures like the FM-index.

The magic of the BWT lies in its ability to group identical characters together while retaining the original string's information, albeit in a transformed format.

// Detect dark theme var iframe = document.getElementById('tweet-1873089359106458098-729'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1873089359106458098&theme=dark" }

Let’s explore how BWT is applied step-by-step using the example banana$.

Applying BWT on `banana$`

Start with a Sentinel Character Append a sentinel character $ at the end of the string. This character:
- Marks the end of the string.
- Is lexicographically smaller than all other characters.

For banana, we get banana$.

Generate All Rotations Create all cyclic rotations of the string:

   banana$
   anana$b
   nana$ba
   ana$ban
   na$bana
   a$banan
   $banana

Sort the Rotations Lexicographically After sorting the rotations:

   $banana
   a$banan
   ana$ban
   anana$b
   banana$
   na$bana
   nana$ba

Extract the Last Column The last column of the sorted rotations forms the BWT:

   BWT: annb$aa

Fun Story :

One of the best parts about attending such Winter schools are the people around you. For example, this is the gossip between me and Alfiya xD.

Yes this is real and was a meme throughout the winter school lmao.

Reversibility: Going Back to the Original Text

One of the most fascinating aspects of BWT is its reversibility. You can reconstruct the original string from the transformed version without losing any information. Here’s how:

Sort the BWT Create the first column by sorting the characters of the BWT. For annb$aa, the sorted order is:

   $aabnn

Link Columns The relationship between the first and last columns allows us to reconstruct the string. By tracking character positions and their order, we recover banana$.

I think my notes will make more sense here

This lossless transformation ensures that no data is discarded during the process.

Succinct Indexing with BWT

The succinct index created by BWT is used in data structures like the FM-index, which powers efficient string searching. Instead of storing the entire text, the FM-index enables queries by leveraging the BWT and auxiliary data structures like suffix arrays. This is especially valuable for applications requiring high-speed pattern matching.

BWT and Suffix Trees

In class, Prof. Chirag Jain explained how BWT interacts with suffix trees and suffix arrays:

Suffix Trees:

By applying BWT to suffix trees, we can reduce memory usage while maintaining efficient query performance. The BWT clusters repeated substrings, which aligns closely with how suffix trees group similar patterns.
Genome Indexing:

In bioinformatics, the BWT is critical for indexing and searching genomic data. Tools like Bowtie and BWA rely on the FM-index (a BWT-based structure) to align DNA sequences efficiently.

Why is BWT Useful for Genomics?

In genome sequencing, researchers often need to match short DNA sequences against massive genomic datasets. The BWT-based FM-index enables:

Space Efficiency: Compact representation of the genome.
Fast Searching: Efficient pattern matching without decompressing the data.
Scalability: Handles terabytes of genomic data.

For example, if a query DNA sequence is AGCT, the BWT groups occurrences of A, G, C, and T together, making it faster to locate matches.

Conclusion

The Burrows-Wheeler Transform is not just an algorithm—it’s a gateway to understanding the interplay between mathematics and computer science. From powering compression tools like bzip2 to optimizing bioinformatics pipelines, BWT continues to revolutionize how we process and store data.

Attending ACM Winter School 2024 gave me a fresh perspective on how these elegant algorithms shape our world. If you’re interested in exploring string algorithms, BWT is the perfect starting point, right after Suffix Trees and Suffix Trees. Also I'll be willing to sell my winter school notes, the bidding starts at 69 bucks xD.

A Lab Manual to Devops

Akash Singh — Thu, 19 Dec 2024 18:35:08 +0000

Wait a lab manual?

As it stands, there is an upcoming exam on devops for some very good friends of mine. And with the lack of material provided by college combined with incompetent faculties, this subject quickly became a nightmare for them.

Fear not, since I was summoned to help out this crisis. Time to put some respect on the name of devops and in the process, try and make the life of some of my friends easier.

Note : This handout will contain solutions and explanation for all 6 of the following questions:

This guide is written not only to help the people who are appearing for Course Code 22CBL54 , but for anyone who wants to get into devops in general.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

What is DevOps Again?

DevOps is the buzzword you can't escape, and for good reason. It's a blend of Development and Operations—a methodology aimed at **streamlining the software development lifecycle (SDLC) by improving collaboration between developers and IT operations teams.

Think of DevOps as a recipe where:

Dev = Building the dish (coding)
Ops = Serving it piping hot to hungry users (deployment and maintenance)

Why Should You Care?

Speed: Faster delivery of features.
Collaboration: No more blaming each other—“It works on my machine” doesn’t fly here.
Reliability: Fewer bugs in production (touch wood).
Scalability: Handle user traffic like a boss.
Security: Automate boring-but-important security checks.

Hope that makes sense? Basically if the app crashes on production, first they will come eat my head and ask me what is wrong. As a devops engineer, since I have setup the pipeline for the deployment, I should be able to look at the monitoring tools or the logs or the infra and tell them okay this is the guy who fucked up (9/10 times its the developer, 1/10 times I done the goof while setting up the infra or pipeline hehe)

Okay I hope it makes sense now, basically we devops engineers are a cross breed and think we are the main characters.

Okay now the premise is set, lets get to passing this lab internals ay?

Question 1: GIT Operations

Problem Statement

To perform Git operations like creating a repository and fetching, pulling, and branching.

Solution

Step 1: Install Git

Use your terminal like a pro:

sudo apt update
sudo apt install git

Step 2: Create a Repository

Navigate to your project folder:
```
cd ~/your_project
```
Initialize Git:
```
git init
```

Add and commit files:

git add .
git commit -m "Initial commit"

Step 3: Fetch and Pull

Fetch brings changes from the remote repository but doesn't merge them:
```
git fetch origin
```
Pull fetches and merges changes:
```
git pull origin main
```

Step 4: Branching

Create a new branch:
```
git branch feature-branch
```
Switch to it:
```
git checkout feature-branch
```

Merge back to main:

git checkout main
git merge feature-branch

In general, pretty self explanatory program. I hope the person reading this has once in their life used git . Is that too much to ask from 3rd year students? Wdym get a life Sky?

Question 2: The DevOps Lifecycle

The DevOps lifecycle involves a series of stages—Code, Build, Test, Configure, and Monitor—designed to streamline the software development process.

Code: Writing and managing code collaboratively.
Build: Compiling source code into executable artifacts.
Test: Automating testing to ensure code quality.
Configure: Setting up the environment for deployment.
Monitor: Tracking application health in production.

Alright, we will build a very small web app that displays "Hello, Devops!" and then implement the above 5 steps in it.

This is kind of a very vague question and a million ways to implement and answer this. I will try to give the simplest solution that should ideally fetch the full marks or I delete my Linkedin.

Step 1: Code

Create the Flask Application
File: app.py

from flask import Flask    
app = Flask(__name__)

@app.route('/')
def home():
    return "Hello, DevOps!"

if __name__ == "__main__":
    app.run(host='0.0.0.0', port=5000)

Set Up a Virtual Environment

python3 -m venv venv
source venv/bin/activate  # Linux/Mac
venv\Scripts\activate  # Windows

pip install flask

pip freeze > requirements.txt

Run the Application
python app.py
Open in your browser: http://localhost:5000.

Step 2: Build

If you know, Python is an interpreted language. There is no build step so to speak, since the code gets executed line by line. So here, build refers to the preparation of the environment and making sure the app is ready to run.

We will write a simple shell script (build.sh) to install dependencies and ensure the app runs:

#!/bin/bash
echo "Setting up environment..."
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
echo "Environment setup complete. Run the app with: python app.py"

Run the script:
```
bash build.sh
```

Step 3: Test

Now since the program demands we write some test, we will write a very simple test that checks if the app is running or not. Btw shameless plugin but if you want to write tests with 0 code, checkout my Keploy Vscode Extension xD

**Test File (test_app.py):

import unittest
from app import app

class TestApp(unittest.TestCase):
    def test_home(self):
        tester = app.test_client(self)
        response = tester.get('/')
        self.assertEqual(response.status_code, 200)
        self.assertEqual(response.data, b"Hello, DevOps!")

if __name__ == "__main__":
    unittest.main()

Run the tests:
```
python -m unittest test_app.py
```

Step 4: Configure

Configuration here focuses on making sure the app can run in different environments. One simple approach is to use environment variables.

Create a .env file:
```
FLASK_PORT=5000
```

Modify app.py to read the environment variable:

import os
from flask import Flask

app = Flask(__name__)

@app.route('/')
def home():
    return "Hello, DevOps!"

if __name__ == "__main__":
    port = int(os.getenv("FLASK_PORT", 5000))
    app.run(port=port)

Use python-dotenv to automatically load .env files:
```
pip install python-dotenv
```

Add to app.py:

from dotenv import load_dotenv

load_dotenv()

Step 5: Monitor

Now monitoring is very very complicated when you bring in fancy tools like Prometheus , Grafana, Loki etc. Implementing all of that is beyond the scope of this beginner's guide . Monitoring in general means just seeing what the app is doing while it is running , like logs, health , any errors etc. We will integrate some basic monitoring without external tools:

Add a /health endpoint to check if the app is running:

@app.route('/health')
def health():
    return {"status": "up"}, 200

Log requests to a file:

import logging

logging.basicConfig(filename='app.log', level=logging.INFO)

@app.before_request
def log_request():
    logging.info(f"Request: {request.method} {request.path}")

Now when the faculty asks where monitoring, show him the localhost:5000/health path and the app.log file. Let him be happy because again, things can get very complicated here if we want them to be fancy.

So what exactly did we do?

Code: Python Flask app built with basic coding practices.
Build: Prepared the environment with a venv and a build script.
Test: Used Python’s unittest to verify the app’s behavior.
Configure: Used .env and python-dotenv for managing environment variables.
Monitor: Added a /health endpoint and request logging.

And that’s how we do DevOps the easy way! 🎉

Question 3: Continuous Integration with Jenkins

Now what's this new circus Jenkins? Remember when i said that as a devops guy, I have to setup the pipeline to ensure the app / code gets deployed? Yes so that pipeline is usually built using github actions and in fancier companies, they use this tool called Jenkins

Steps to Install Jenkins :

This should already be done in your lab system, if not tell your faculty that hey jenkins not installed or look up the installation instructions online , don't memorize this. Only the command to start jenkins.
Remember you need an existing app to deploy it using jenkins, for simplicity I am using the same python app we built in Q2. It needs to be pushed to github as you'll need to connect the pipeline to your repo.

Install Java (required for Jenkins):

sudo apt update
sudo apt install openjdk-11-jdk

Add the Jenkins Repository:

curl -fsSL https://pkg.jenkins.io/debian/jenkins.io.key | sudo tee \
/usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
https://pkg.jenkins.io/debian binary/ | sudo tee 
/etc/apt/sources.list.d/jenkins.list > /dev/null

Install Jenkins:

sudo apt update
sudo apt install jenkins

Start Jenkins and Check Status:

sudo systemctl start jenkins
sudo systemctl status jenkins

Configure Jenkins

Access Jenkins:

Open a browser and go to http://:8080 or http://localhost:8080.
Unlock Jenkins:

Use the admin password found in: (cant memorise , the password is unique always, you'll find it here)
```
cat /var/lib/jenkins/secrets/initialAdminPassword
```
Install Recommended Plugins:

Follow the on-screen setup to install necessary plugins (e.g., Git, NodeJS, Pipeline).

Jenkins Pipeline for a Web Application

Create a New Job:

-   Go to Jenkins Dashboard and click *"New Item"*.
-   Choose *"Pipeline"* and give it a name.

Configure Git Repository:

-   In the Pipeline configuration, specify the repository URL:
    -   *Source Code Management* > Select *Git* > Enter your repository URL.
-   Add credentials if needed.

Define the Pipeline Script:

pipeline {
    agent any

    environment {
        VIRTUAL_ENV = 'venv' // Virtual environment directory
        FLASK_PORT = '5000' // Port to run the Flask app
    }

    stages {
        stage('Clone Repository') {
            steps {
                echo 'Cloning the repository...'
                git branch: 'main', url: 'https://github.com/ahanasrinath/myflaskapp.git'
            }
        }

        stage('Set Up Environment') {
            steps {
                echo 'Setting up virtual environment and installing dependencies...'
                sh '''
                python3 -m venv $VIRTUAL_ENV
                source $VIRTUAL_ENV/bin/activate
                pip install -r requirements.txt
                '''
            }
        }

        stage('Run Tests') {
            steps {
                echo 'Running unit tests...'
                sh '''
                source $VIRTUAL_ENV/bin/activate
                python -m unittest discover -s . -p "test_*.py"
                '''
            }
        }

        stage('Build and Package') {
            steps {
                echo 'Packaging the application...'
                sh '''
                tar -czf flask-app.tar.gz app.py requirements.txt .env
                '''
            }
        }

        stage('Deploy') {
            steps {
                echo 'Deploying the application...'
                sh '''
                # Copy the application to the deployment server
                scp flask-app.tar.gz user@your-server:/path/to/deployment

                # SSH into the server and deploy
                ssh user@your-server << EOF
                cd /path/to/deployment
                tar -xzf flask-app.tar.gz
                source $VIRTUAL_ENV/bin/activate
                nohup python app.py > app.log 2>&1 &
                EOF
                '''
            }
        }
    }

    post {
        success {
            echo 'Pipeline completed successfully!'
        }
        failure {
            echo 'Pipeline failed. Check the logs for more details.'
        }
    }
}

NOW HERE IS THE THING.....Since we dont have a server to begin with, there is no place for jenkins to deploy our app so to speak. Which again makes me wonder why the fuck have they told y'all to deploy something without any context whatsoever as to where it should be deployed?

Will the college give yall a dedicated server to deploy to? Very unlikely. Do they even know that they need to do so? Also unlikely. So take the last deploy stage with a grain of salt.

Hope this is not y'all in exam hall, its me everyday.

NOTE:

If they give you a random app to deploy with jenkins, *DO NOT PANICK. You'll realise whatever steps we have written in the jenkins file are the actual commands we executed in Q2. So, they can be replaced with any specific set of instructions for any specific app/ tech stack.*

For Ex:

If they give you a react app, the setup environment would be replaced with just npm install and the build will be replaced with npm run build. Similarly for java app , it might be something different like javac app.java or gcc main.c for a c file.

Point is not to panick, they dont know what they are teaching or the questions they are giving. As long as you write the steps more or less like the above, they will know that you have cooked something.

Question 4: Version Control Tools

Git is the king here, but alternatives like CVS, RCS, and Mercurial deserve mention. Each has its pros and cons:

RCS: For single-file projects.
CVS: Old-school, great for teams.
Mercurial: Git’s less famous sibling, preferred by projects like Mozilla.

Chalo , lets first install these (hopefully they are already installed in the lab computers y'all will be using). Use the brew commands for Mac and apt for linux.

RCS

sudo apt install rcs

brew install rcs

CVS

sudo apt install cvs

brew install cvs

Git

sudo apt install git

brew install git

Mercurial

sudo apt install mercurial

brew install mercurial

And here are the basic basic commands for all of them:

1. Revision Control System (RCS)

# Initialize RCS directory
mkdir RCS

# Check-in a file to create its first revision
ci -u filename

# Check-out a file for editing
co -l filename

# View version history
rlog filename

2. Concurrent Versions System (CVS)

# Initialize a repository
cvs -d /path/to/repo init

# Add a project to CVS
cvs import -m "Initial import" project_name vendor_tag release_tag

# Checkout a project
cvs checkout project_name

# Commit changes
cvs commit -m "Commit message"

3. Git

# Initialize a repository
git init

# Add files to the repository
git add .

# Commit changes
git commit -m "Initial commit"

# View history
git log

# Push changes to a remote repository
git remote add origin <repository-url>
git push -u origin main

4. Mercurial

# Initialize a repository
hg init

# Add files to the repository
hg add

# Commit changes
hg commit -m "Initial commit"

# View history
hg log

# Push changes to a remote repository
hg push <repository-url>

Question 5: Docker Basics

Okay little context is required as to what this Docker is. Docker is the jaan of every Devops engineer. Without Docker, there is only chaos and suffering and pain.

What is Docker?

Docker is an open-source platform that allows you to develop, ship, and run applications inside containers. Think of a container as a lightweight, portable, and consistent environment that contains everything your application needs to run—code, libraries, dependencies, and even the runtime.

Why Use Docker?

Basically to solve the problem of "ye code mere system pe chal raha h, but not on your system". A docker container simulates the entire environment and all the dependencies, so wherever you run the container, it will run reliably and without any circus. Now you dont have to fight that rich friend who has a mac or that nerd who lives only on linux. Kahi bhi code likho, chalega ekdam aram se using Docker.

Core Concepts in Docker

Docker Image:
- A blueprint for a container.
- Contains the app code, libraries, and dependencies.
- Example: The image we will build using the Dockerfile.
Docker Container:
- A running instance of a Docker image.
- Isolated, lightweight, and portable.
Dockerfile:
- A text file containing the instructions to build a Docker image.
- Example:
```
FROM python:3.8-slim
WORKDIR /app
COPY . .
CMD ["python", "app.py"]
```
Docker Hub:

-   A cloud-based registry for sharing Docker images.
-   Think of it as "GitHub for containers."

Docker is widely used in DevOps because it:

Simplifies app deployment.
Works seamlessly with CI/CD pipelines.
Allows for microservices architecture, where each service runs in its own container.

Highly recommend watching this video by Fireship on Docker and this one by Coderized

IN VERY SIMPLE TERMS : Imagine you're packing for a trip:

The Docker Image is the suitcase, pre-packed with all the essentials (toothbrush, clothes, etc.).
The Docker Container is you actually carrying the suitcase on your trip.
Docker Hub is the store where you buy pre-packed suitcases (images).

Coming back to our lab program, how do we dockerise our python app? First download Docker Desktop from here

Now they have mentioned that you should run different container OS, which is brain dead easy:

1. Pull Operating System Images

Docker Hub provides a wide variety of operating system images.

Search for an OS image:

docker search <os-name>

Example for Ubuntu:

docker search ubuntu
Pull an OS image:

docker pull <os-name>:<tag>

Example for Ubuntu:

docker pull ubuntu:latest

2. Run Containers from the OS Image

Run an interactive container:

docker run -it <os-name>:<tag> /bin/bash

Example:

docker run -it ubuntu:latest /bin/bash

This will give you a shell prompt inside the container.
List running containers:

docker ps
List all containers (including stopped ones):

docker ps -a
Stop a running container:

docker stop <container-id>
Remove a container:

docker rm <container-id>

3. Building a Custom Application Container

Now we will dockerise your flask app! The above step 2 is not necessary for this step.

Create a Dockerfile in the root dir

FROM python:3.8-slim

WORKDIR /app

COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Build the Docker Image

docker build -t flask-devops-app .
Run the Container

docker run -d -p 5000:5000 flask-devops-app
Access the app at http://localhost:5000.

I will still write what each step does, feel free to skip if you can just memorise it:

FROM python:3.8-slim

Purpose: This specifies the base image for your container.
- python:3.8-slim is a lightweight version of the Python 3.8 image. It contains only the essential components required to run Python, making it smaller and faster to download.

WORKDIR /app

Purpose: This sets the working directory inside the container to /app.
- Any subsequent commands like COPY or RUN will execute relative to this directory.
- It ensures your application code and dependencies stay organized in a specific directory within the container.

COPY requirements.txt requirements.txt

Purpose: This copies the requirements.txt file from your local system (host) to the /app directory inside the container.
- The requirements.txt file contains all the Python libraries your app depends on.

RUN pip install -r requirements.txt

Purpose:

This installs all the Python dependencies specified in the requirements.txt file using pip.
- For example, if your app uses Flask, the requirements.txt might contain:
  
  flask==2.1.2

COPY . .

Purpose: This copies all files from your local working directory (host) to the /app directory inside the container.
- It ensures your app.py and any other necessary files (e.g., templates, static assets) are available in the container.

CMD ["python", "app.py"]

Purpose: This specifies the command the container should run when it starts.
- Here, it tells Docker to execute python app.py, which starts your Flask application.
- The CMD command runs the app in the foreground to keep the container alive while your app is running.

THATS IT! YOUR DOCKER APP IS NOW ALIVE!!!

4. Manage Docker Containers and Images

Some extra commands because they maybe useful yk

View all images:

docker images
Remove an image:

docker rmi <image-id>

Stop and remove all containers:

docker stop $(docker ps -aq)
docker rm $(docker ps -aq)

Prune unused data (stopped containers, networks, etc.):

docker system prune -a

Question 6: Deploying Apps with Docker

Congrats! If you have reached this far I am v. v. proud of you! Here is your reward: You dont have to study Question 6. Yes, question 5 and 6 are literally same to same steps! You can deploy the same flask app using the same step 3 here as well. For safety, here is the java implementation as well, you will need a simple hello world written in java btw, here it is:

Create Main.java in your project directory

public class Main {
    public static void main(String[] args) {
        System.out.println("Hello, Dockerized Java App!");
    }
}

Then add the Dockerfile in the same folder :

Dockerfile for a Java App

FROM openjdk:11
COPY . /app
WORKDIR /app
RUN javac Main.java
CMD ["java", "Main"]

Explanation for the steps here are the same as the Dockerfile for our Flask app, only that instead of python commands we use java commands:

FROM openjdk:11

Purpose: This specifies the base image for your container.
- openjdk:11 is a lightweight image containing the Java Development Kit (JDK) for Java 11.
- It provides everything needed to compile and run Java applications.
- Using a specific version (11) ensures consistency and avoids issues from version mismatches.

COPY . /app

Purpose: This copies all files from your current working directory on the host machine to the /app directory inside the container.
- For example, this will include your Java source files (Main.java) and any other files needed to run your application.

WORKDIR /app

Purpose: This sets the working directory inside the container to /app.
- All subsequent commands (like RUN or CMD) will execute relative to this directory.
- It ensures your application files are in the right location before compilation or execution.

RUN javac Main.java

Purpose: This compiles your Main.java file using the javac (Java Compiler) command inside the container.
- The javac command generates the bytecode .class file (e.g., Main.class), which is necessary to run the Java application.
- If your application has multiple Java files or dependencies, make sure they are all copied and accessible for compilation.

CMD ["java", "Main"]

Purpose: This specifies the command the container should run when it starts.
- java Main executes your compiled Java program (Main.class).
- CMD runs the application in the foreground, ensuring the container remains active while the application runs.

After building and running the container, your Java application will execute and output results to the terminal (or wherever your app is programmed to send them).

Build and Run

Build the image:
```
docker build -t my-java-app .
```
Run the container:
```
docker run my-java-app
```
Output:

You should see:

Hello, World!!

Conclusion

I hope this blog helps y'all out in passing this lab exam. This was genuinely fun to write and solve , not sure what your faculty did the entire semester if he/she didn't share this material. While all of this is available online and probably written and explained better, I tried my best in explaining it in my terms and how I would solve all of them.

Then again, your lab systems will have internet. Without internet, you can't run any of the above programs anyways? So yes , its easy to google up the steps or look up this handout during the exam.

Feeling grattitude? I accept payments in the form of cadbury fuse , mango lassi and ocassionally *ganne ka juice. Yes, this is part of my work at Vance and as a devops engineer. Yes, this is a very very smol drop in the ocean of devops and things get complicated really quickly with monitoring and kubernetes and yamls etc. Yes I need to touch some grass and yes, all of you reading are *profoundly welcome <3

Thanks for reading :))

Why we Built a Mini-Language for a Golang Hackathon

Akash Singh — Fri, 29 Nov 2024 07:33:01 +0000

A Hackathon, Again?

At this point, I have been to 9 hackathons, one of them being an international one, even winning at 4 of them. Then again, when my juniors Dhruv and Tushar told me about a Golang Specific hackathon, I dragged Harsh along with us because why not. And not just Harsh, I dragged along 40+ people from our team Point Blank, which ended up making the hackathon our own internal competition haha.

All of us in our team GoGoingGone (lmao) had good experience working with Golang, But we wanted to do more than just build another tool. We wanted to innovate. That’s when the idea struck—let's build a mini-language to define dynamic, configurable data pipelines.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

Introducing Fractal

Fractal started as a data processing tool for seamless migration from legacy systems (like SQL databases and CSV files) to modern platforms such as MongoDB or AWS S3. But we wanted more than just another ETL tool. The idea was to make it highly flexible and user-friendly, allowing users to define validation and transformation rules with a simple, declarative syntax—a mini-language within the tool.

Why a Mini-Language?

We observed that most tools in the data pipeline space rely on rigid configurations or custom scripts. This approach often requires significant programming expertise, which limits accessibility for non-developers. A declarative mini-language provides:

Simplicity: Users define rules in an intuitive, human-readable format.
Flexibility: It accommodates a wide range of use cases, from basic validations to complex transformations.
Scalability: The mini-language can evolve as new requirements arise.

This mini-language wasn’t about reinventing the wheel—it was about providing an abstraction to streamline data transformations and validations.

When this is combined with a simple yaml file configuration, we thought we hit the mark of making a easy to configure data pipeline that can process data from one source to another on scale.

The Core: Validation and Transformation Syntax

We designed the syntax to be simple yet expressive, focusing on two primary operations:

Validation Rules These ensure incoming data meets specific quality standards before further processing. For example:

   FIELD("age") TYPE(INT) RANGE(18, 65)
   FIELD("email") MATCHES(EMAIL_REGEX)
   FIELD("status") IN ("active", "inactive")

Transformation Rules These enable data enrichment or restructuring. For example:

   RENAME("old_field", "new_field")
   MAP("status", {"0": "inactive", "1": "active"})
   ADD_FIELD("processed_at", CURRENT_TIME())
   IF FIELD("age") > 50 THEN ADD_FIELD("senior_discount", TRUE)

This abstraction allowed users to process diverse datasets with minimal effort, enhancing productivity and reducing complexity.

In the middle of figuring out how to make the lexer and parser of this language, the team at GoFr.dev took us all upstairs for a stress busting session, which was full of late night sharayis and jam sessions!

Building Fractal at the Hackathon

The hackathon wasn’t just about creating the mini-language. We also had to build the surrounding infrastructure, ensuring Fractal was:

Extensible: Supporting multiple input/output formats like JSON, CSV, SQL databases, and message queues.
Configurable: YAML-based configuration for defining pipeline workflows, integrating the mini-language seamlessly.
Robust: Handling errors gracefully with options like LOG_AND_CONTINUE or STOP.

We divided the work into four modules:

Mini-Language Implementation: Designing the lexer and parser to interpret the custom syntax.
Data Integrations: Adding support for common data sources and destinations.
Pipeline Engine: Orchestrating validation, transformation, and error handling.
CLI Interface: Providing a simple interface for defining and running pipelines.

Challenges We Faced

Designing the Syntax Striking a balance between simplicity and flexibility was a challenge. We iterated multiple times to finalize the syntax.
Building the Parser Implementing a custom lexer and parser in Golang was time-consuming but rewarding.
Real-Time Feedback Ensuring that the mini-language provided meaningful error messages to guide users was critical for usability.
Time Constraints Building a tool of this scale in a hackathon setting required precise planning and seamless coordination.

And what happened after that?

Despite our strong showing at the GO for GOFR hackathon, we faced a critical challenge during the final evaluation. The judges requested a live demonstration in addition to our recorded demo, and unfortunately, we encountered an unexpected bug in our parser logic during the live run. Given the complexity of building a robust custom parser within just 24 hours, it was an ambitious feature to develop, and while our recorded demo showcased its functionality, achieving 100% accuracy under time constraints proved difficult. This hiccup ultimately cost us the top prize. However, our efforts were still highly regarded, and our team's clear vision and compelling delivery earned us the honor of "Best Pitch," highlighting our potential and ingenuity.

So Hackathons huh?

Hackathons are often about pushing boundaries and exploring uncharted territories. Fractal was our attempt to redefine how data processing tools can work—by making them accessible, modular, and developer-friendly.

I couldn't have asked for a more likeminded set of people to work with me on this, absolute best and hardworking teammates without a shadow of a doubt. Looking forward to what brings me to my next hackathon, dare I say, A RUST based hackathon? xD

Check out Fractal on GitHub

SkySingh04 / Fractal

A flexible, configurable data processing tool

Fractal

Fractal is a flexible, configurable data processing tool built with GoFr and Golang. Fractal is designed to handle data ingestion from multiple sources, apply powerful transformations and validations, and deliver output to a wide range of destinations. With Fractal, you can automate complex data workflows without needing to manage low-level details Here's the documentation for setting up a new integration in your project:

Custom Syntax Documentation for Validation and Transformation Rules

1. Overview

The custom syntax enables users to:

Validate incoming data to ensure it meets predefined conditions.
Transform data fields to fit desired formats, structures, or requirements.
Define flexible error-handling strategies for data processing pipelines.

Rules can be written for any data source or destination, such as JSON, YAML, CSV, SQL Databases, Message Brokers, or Cloud Services.

2. Validation Rules

Validation rules ensure that data meets specific quality and integrity requirements.

…

View on GitHub

Pitch Deck : Drive Link

Or Try it Yourself and let us know what do you think!

How Talking to a College Senior taught me Devops/Gitops

Akash Singh — Fri, 22 Nov 2024 17:20:10 +0000

The Story

I love going to hackathons. Not just for participating, but for meeting new people and seeing what people are building in general. This year in May, I went to a hackathon Aventus, not for participating but for meeting some seniors and peers. While there, I met Rishabh Lakhotia, an alumnus of Point Blank , who told me of an intern opening at his company Vance. Now I personally always had an interest for Devops, but when I talked to him, I was bombarded with words like GITOPS and Infrastructure As Code and Terraform! The only course of action was to figure out what all of this meant so that I could have a chance clearing the interview. The best way I figured I could stand out was implementing all of what I learnt in my existing project. This blog post will break down the essentials of GitOps and share insights from my implementation process.

Introduction

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

What is GitOps?

At its core, GitOps brings Git’s version control capabilities to the world of DevOps. By treating Git as a "single source of truth," GitOps manages both application code and Infrastructure as Code (IaC) in separate Git repositories. This setup provides several benefits:

Easy Rollback: Git history enables version tracking, allowing teams to revert to a previous state in seconds if something goes wrong.
Increased Security: By controlling code changes through pull requests and Git’s in-built security features, GitOps enhances deployment security.
Complete Automation: With automated CI/CD pipelines, deployment becomes a smooth, consistent process where each code push triggers relevant updates and builds.

In GitOps, the desired state of the infrastructure and applications is stored in Git. Any changes to this state, whether application code updates or infrastructure configurations, are tracked, reviewed, and managed through pull requests. This approach reduces manual intervention and allows teams to deliver faster and more reliable updates.

While learning about Gitops, I found this extremely helpful video

Implementing GitOps in HaalSamachar

Okay now I understood what Gitops is, how exactly do I implement it? Well, lets break it down :

Step 1: Separating Application and Infrastructure Code

To fully leverage GitOps principles, I separated application and infrastructure code into two distinct Git repositories. This organizational shift made it easier to manage each part independently while aligning with GitOps best practices. For HaalSamachar, this meant dedicating one repository to the application codebase and another to the Infrastructure as Code (IaC) scripts.

The benefit of this separation became apparent when making infrastructure updates. Now, I could manage infrastructure changes without interfering with the main application codebase, reducing complexity and providing a more modular approach to handling deployments and updates.

Here is the GitOps Application Repository

SkySingh04 / Haalsamachar-app

Application Repository for HaalSamachar consisting of Backend Microservices built with GoLang including a GraphQL API built using gqlgen and four REST APIs built using Gin and frontend built with NextJs+TypeScript with PostgreSQL powered database, containerized using Docker.

HaalSamachar Application Repository : Consists of Backend Microservices built with GoLang including a GraphQL API built using gqlgen and four REST APIs built using Gin and frontend built with NextJs+TypeScript with PostgreSQL powered database, containerized using Docker using Dockerfiles and CI/CD pipeline configurations.

HaalSamachar Infrastructure Repository : Contains Terraform scripts, Kubernetes manifests, and GitOps configurations for Haalsamachar App.

Features

GraphQL API: Utilizing gqlgen for creating a GraphQL server to efficiently query and manipulate data.

REST APIs: Three REST APIs are built using Gin for handling various functionalities.

Docker & Kubernetes: Containerized using Docker.

Next.js with SSR: Frontend developed using Next.js for server-side rendering (SSR) along with TypeScript and Tailwind CSS.

PostgreSQL: Utilized as the database to store and manage data efficiently.

Firebase Auth: Integrated Firebase authentication for user authentication and authorization.

Continuous Integration/Continuous Deployment (CI/CD)

CI/CD pipelines automate the process of testing and deploying code changes. HaalSamachar utilizes CI/CD practices…

View on GitHub

And Here is the GitOps Infrastructure Repository

SkySingh04 / Haalsamachar-infra

Haalsamachar IAC : Contains Terraform scripts, Kubernetes manifests, and GitOps configurations for Haalsamachar App

HaalSamachar Infrastructure Repository : The Haalsamachar App's infrastructure is managed through an Infrastructure as Code (IaC) approach, incorporating Terraform scripts, Kubernetes manifests, and GitOps configurations. This ensures automated, scalable, and consistent deployment of resources.

HaalSamachar Application Repository : Consists of Backend Microservices built with GoLang including a GraphQL API built using gqlgen and four REST APIs built using Gin and frontend built with NextJs+TypeScript with PostgreSQL powered database, containerized using Docker using Dockerfiles and CI/CD pipeline configurations.

Setting Up Kubernetes Cluster

The kubernetes deployment configuration yaml files are located in /deployment directory. These can be modified to scale the number of pods and other configurations as per requirements.

To deploy HaalSamachar using Kubernetes, follow these steps:

Install Kubernetes: Set up a Kubernetes cluster on your preferred cloud provider or locally using Minikube.
Apply Manifests: Use kubectl apply /deployments command to apply the Kubernetes manifests and deploy the HaalSamachar application to…

View on GitHub

Step 2: Leveraging AWS Services – ECR and ECS

Now two other things that I gathered from my conversation with Rishabh Bhaiya was they are using AWS ECR and AWS ECS. I had no idea what these were but oh well, time to implement them.

In a nutshell :

ECR serves as a private repository where Docker images are stored, ensuring each build is securely stored and readily available for deployment.
ECS handles the orchestration and management of these containers, simplifying the deployment and scaling of containers across a fleet of machines.

Okay, now I need to figure out how to deploy HaalSamachar’s application in a containerized environment.

Step 3: Setting Up ECR and ECS for HaalSamachar

Once I got the basics of ECR (Elastic Container Registry) and ECS (Elastic Container Service) down, I moved on to implement them in HaalSamachar. Here’s how it went down:

Configuring ECR

First, I needed a private, secure location to store the Docker images of HaalSamachar. ECR was perfect for this, as it integrates seamlessly with other AWS services and provides a safe, centralized storage for my Docker images.

Create the ECR Repository: Using the AWS Management Console, I set up a new repository in ECR, allowing it to hold the Docker images for each deployment version of HaalSamachar.
Set Up Permissions: Next, I configured permissions to allow ECS (Elastic Container Service) to pull images from ECR whenever needed.
Push Docker Images to ECR: Every time I update the application, I generate a new Docker image and push it to ECR using AWS CLI. This versioning lets me maintain consistency and track changes efficiently.

Deploying with ECS

With the Docker images stored in ECR, I moved on to ECS, AWS’s container orchestration service. Here’s how I leveraged ECS to deploy and manage HaalSamachar:

Task Definition: ECS required a task definition that specified how my application should run in a containerized environment. I defined details like container image source (pointing to ECR), memory, CPU requirements, and port mappings.
Service Setup: I created an ECS service to manage the deployment and scaling of my container. The service enables ECS to monitor the health of containers and replace any failing instances automatically.
Cluster and Deployment: Finally, I launched the service within an ECS cluster, which facilitated the management of container instances on AWS infrastructure.

By setting up ECS, I didn’t need to worry about manually handling containers. AWS took care of the orchestration, and ECS's auto-scaling capabilities ensured that the application could handle varying traffic loads without manual intervention.

Step 4: Automating Deployments with GitHub Actions

Once the infrastructure was in place, I needed a way to automate deployments. Enter GitHub Actions, a CI/CD tool that was critical in implementing GitOps for HaalSamachar. As it turns out, this was also how Vance handled their workflow.

I set up a workflow in GitHub Actions with the following stages:

Build Stage: Every time I pushed changes to the main branch, the workflow would kick off a build. This stage created a Docker image of HaalSamachar from the latest code.
Push to ECR: After the Docker image was built, it was automatically pushed to the ECR repository. This step ensured that the latest code changes were available in the image repository for deployment.
Deploy on ECS: The final stage involved updating the ECS service with the new Docker image. GitHub Actions triggered the deployment on ECS, which fetched the latest image from ECR and deployed it seamlessly.

This workflow significantly streamlined my deployment process. Each push to the main branch automatically triggered a full deployment pipeline, reducing the chance for human error and increasing efficiency.

What's the point tho?

My conversation with Rishabh bhaiya was a turning point. Before that hackathon, I only had a high-level curiosity about DevOps. Concepts like GitOps, IaC (Infrastructure as Code), and Terraform felt out of reach, almost like advanced topics for “real” engineers. But Rishabh’s words inspired me to dive in, to experiment with these concepts hands-on and push my project, HaalSamachar, to new heights.

After that initial chat, I studied, broke down, and pieced together everything I could about GitOps and DevOps fundamentals. I watched tutorials, read documentation, and learned by building. Even though I didn’t land the internship at Vance, the journey to prepare for that opportunity reshaped my perspective on software engineering. It wasn’t about the end goal of landing a position but rather the growth I experienced by stretching myself beyond my comfort zone.

Through this project, I realized that HaalSamachar, while perhaps not my most sophisticated work, holds special meaning. It's a project where I could test my understanding of GitOps and learn AWS services like ECR and ECS from scratch. Watching it come together made the late nights of debugging, building Docker images, and learning CI/CD feel incredibly rewarding. In the end, HaalSamachar taught me that you don’t have to wait for the “perfect” project or opportunity to dive into new tech. Just start where you are, learn, and build something—no matter how small, it will move you forward.

As I look to future projects, HaalSamachar will always hold a unique place in my journey. It’s not just a news aggregation tool; it’s the project that introduced me to GitOps and sparked my journey into the world of DevOps. And that’s thanks to a simple conversation with a senior who was willing to share what he knew.

My CloudSEK Internship Experience

Akash Singh — Mon, 11 Nov 2024 12:59:59 +0000

This article is about my internship at CloudSEK.

CloudSEK is a cybersecurity company that uses artificial intelligence and machine learning to identify and manage digital threats!

A lot of my seniors at Point Blank were working / had worked at CloudSEK. So, when I got to know CloudSEK is looking for interns I HAD to apply! In this blog post, I will describe the application process, the interview process, and my experience working at this company.

Intro

I am Akash Singh, a third year engineering student and Open Source Contributor from Bangalore.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

Selection Process

This was probably one of the most in-depth recruitment process I had ever been a part of.

Application :

I had initially found the open intern position on LinkedIn. After applying there, I had asked my seniors at Point Blank to provide a referral for the same. I ended up working in the same Pod (team) as my senior Aditya. Shortly after which, I received a confirmation email and my interview was scheduled.

HR Round :

This round was conducted to ensure I will be able to provide the necessary commitment of 3 months to the internship, along with general discussions about my past internships and experience and why I wanted to work at CloudSEK.

Interview Round 1 :

This round was headed by my team leader. We went over my resume, my past experiences with Golang including my Google Summer of code project. This was followed by some system design questions and then DBMS / SQL fundamentals with a database design problem as well.

If you have a good grasp of CS Fundamentals and Database fundamentals, this round should be easy enough. You will have a positive experience if you are confident in the skills you've listed on your resume. Be thorough with your Projects.

Interview Round 2 :

This round was taken by the tech lead of Core Platform Engineering team. The thing that gave me an advantage in this round was my DNS Server Project
which was written in RUST . The interviewer took keen interest in my project and we discussed it for half an hour. This was then proceeded by a System Design question, which was to be designed and implemented in GO.This really had me sweating and I am actually surprised I was able to implement it.

If you have interesting projects and are thorough with the implementation and various nuances of it, you are good to go! Stay confident and try to solve the system design question while considering all edge cases. The company is okay if you google during the interview.

SELECTED! 🎉

The thing I realised about CloudSEK was, whatever questions / topics they had covered during the interviews, they had direct application during the job. This was a breathe of fresh air from all the leetcode based hiring (xD)

My Experience

Working as a Backend Engineer intern at CloudSEK was a transformative experience where I gained hands-on exposure to advanced development practices, cloud services, and CI/CD pipeline management. My primary focus areas included maintaining and developing microservices for the CloudSEK Community Server, which was entirely written in GO. I tackled various tasks ranging from writing controllers to exploring OpenTelemetry for monitoring, adding both breadth and depth to my skillset, along with designing Databases using POSTGRES and MongoDB (yes now the interview questions make sense).

I worked primarily from office during my internship. The office was always full of snacks, unlimited caffeine and hardworking individuals alike.

I explored various monitoring and observability tools, such as Grafana, Loki, and OpenTelemetry, to set up and analyze metrics effectively. This deep dive helped me understand the importance of real-time insights for backend health, helping us quickly identify bottlenecks and optimize resource allocation. Additionally, I learned so much about Load testing, writing Integration and Unit tests, and fell more and more in love with Golang❤️.

I was also exposed to Gitops and secret managements, writing scalable microservices and monorepos, generating documentation using Swagger and deploying my services using Kubernetes! The fact that I was given the full liberty to design, implement and deploy my own services meant that I had to take care of everything from writing tests to building the CI/CD pipeline to deploying the changes using ArgoCD.

CloudSEK is a RFC-First company. What this means is, before you actually start coding, you are supposed to write a Request for Change (RFC) document for whatever changes you are introducing / service you are building. I had to write many RFC's while working on the Cloudsek Community Server , which used to go through multiple rounds of reviews. This documentation helped align our team on design decisions and ensured clarity on the objectives, benefits, and technical details of upcoming features. Completing the RFCs taught me the value of clear, technical communication in project planning.

Along with all of this learning, I was also exposed to so many AWS tools such as Simple Storage Service (S3), Simple Queuing Service (SQS) along with tools like Kafka, Redis etc. Oh and did I mention, you cannot survive a day in CloudSEK without Docker? Haha, yes I worked with lots of Docker and CI pipelines during my internship and it was super fun!

My Introduction To Cybersecurity

One of the highlights of my internship at CloudSEK was my first real exposure to cybersecurity. I not only had the chance to work on projects directly impacting cybersecurity solutions, but I also completed an internal course, CloudSEK External Threat Monitoring Solutions, which deepened my understanding of the field. This course provided valuable insights into identifying and managing digital threats, leveraging tools and techniques used by cybersecurity professionals to protect organizations against external attacks.

The course wasn’t just theoretical—it included a Capture The Flag (CTF) challenge that put my problem-solving skills to the test. Engaging with real-world cybersecurity scenarios and tackling CTF challenges gave me a hands-on experience that solidified my foundational knowledge in cybersecurity.

Even though cybersecurity was not my field of interest, this exposure only increased my field of knowledge. Thanks to the CPE team for making me do this!

Conclusion

CloudSEK is an amazing place to be a developer. I’m incredibly grateful for the support and camaraderie of my team. Each senior I worked with brought a wealth of knowledge and was always eager to share, making every project not only a task but a learning opportunity. From brainstorming sessions to coding sprints, the team’s dedication and passion for cybersecurity were infectious.

The office itself was a constant source of motivation—with an endless supply of coffee, snacks, and a culture that genuinely valued hard work and curiosity. Truly one of the best offices in Bangalore to work at!

You will learn and grow as the team is open to trying new technologies. They keep on adopting new tools and technologies thus there is always something to learn! I’m excited to take all that I’ve learned here into my future endeavors, carrying with me the invaluable experience of working at a company that’s genuinely making a difference in the digital landscape.

Thanks for reading my article :)

Hacktoberfest : The right way to do open source

Akash Singh — Sun, 03 Nov 2024 22:56:12 +0000

This is a submission for the 2024 Hacktoberfest Writing challenge: Contributor Experience

When October began, Me and some seniors at our club Point Blank were wondering how we can get the new and existing members of our club to contribute to Open Source and Hacktoberfest.

In this blog I will outline and answer the common doubts related to open source contributions and how to make Hacktoberfest worth it for not only you, but the community as well.

Intro

I am Akash Singh, a Backend Developer and Open Source Contributor.
Here is my LinkedIn, GitHub and Twitter

I go by the name SkySingh04 online.

My Contributions

Before I start explaining and answering the FAQ's, It's important to note how I personally did my Hacktoberfest. I was able to finish all 4 PR's, and am currently working on 2 more PR's even after **Hacktoberfest **ended.

You can check out the Merged PR's here : Github Gist and read about my contributions here LinkedIn Post

As far as I scrolled through LinkedIn, I realize this is on the lower end of contributions. This is where I have a very strong opinion regarding open source in general :

Quality Over Quantity

Is it easy enough to submit 10 more Readme contributions? Yes, infact a number of repos on Github are made specifically for the purpose of getting the Hacktoberfest badges, with zero / minimum actual code.

Now I am aware of the guidelines of Hacktoberfest that encourage both Code and No Code Contributions. As such, I am not against No Code Contributions , but I am strongly against low effort contributions, which brings us to one of the most common questions.

How To Avoid Low Effort Contributions?

We live in an era of the internet where making a Readme change is as simple as prompting Chatgpt/LLMs with the right data and it will literally write the Readme/ Documentation for you.

As a rule of thumb, if the issue can be solved via a simple prompt to Chatgpt, are you even making a contribution to open source in the first place?

Moving a step ahead from open source, how is making such a contribution helping you? Are you learning something new? Are you exploring a new codebase? Chances are, the answer is a NO.

Circling back to the question, I believe the best kind of contributions are those that not only are on Production Code and help the maintainers, but also help you learn / explore something new. It maybe a new language you wanted to try, or a new project you find interesting. The only rule is to not waste the maintainers time with low effort questions / contributions.

My whole agenda of doing Hacktoberfest was to get more comfortable with RUST and I am pretty satisfied with my 3/4 PR's being Rust contributions.

How To Find Repositories To Contribute?

I will be sharing the trick I shared with my peers and juniors at Point Blank. Lets assume you want to contribute to Rust based projects.

One of the best places to find repos to contribute is the Github topics page

On this page , we can easily sort repos by the language of choice.

The trick is to sort the repos via recently updated. This ensures you get repos that are not only active, but are actively looking for contributions!

And voila! You now have a list of repos to go find issues to contribute to!

How To Find Issues to work on?

Now, what happened with a lot of my peers at Point Blank (and myself) is that we couldn't find good issues that interest us. And when we did, there was already someone working on it.

Well of course, one solution is to keep on looking for unassigned issues, another is to ask the maintainers if they have any issues for new contributors.

What I want to talk about is this third scenario where someone has committed to solving a issue but never does. Such kind of people not only deserve a special place in hell, but also end up ruining the spirit of open source.

Lets take this issue in Doctl for example.

In this issue, a contributor initially expressed interest in working on it, and the maintainer gave them the go-ahead. However, after stating they would "close it by Monday," no progress was made, nor was any PR submitted by the set deadline.
This left the issue open and unresolved, which can be frustrating for the project’s progress, especially when other contributors were interested in solving it but held off in respect of the initial claim.
After the original assignee didn’t follow through, I revisited the issue, ensured that no one was actively working on it, and eventually submitted the PR myself.

Ultimately, this demonstrates why maintaining clear communication and respecting deadlines are crucial. Committing to an issue and not following through can block other contributors and delay valuable improvements.

Key Takeaway

If you claim an issue, make sure you’re committed to completing it or communicate with the maintainers if you’re unable to continue. Open source relies heavily on collaboration and transparency, so leaving others in the dark harms the project and wastes community effort.

Amaze, Now really quickly let's go over some Do's and Don'ts

Do’s ✅

Find Projects with Active Maintainers and Open Issues ✅
Look for repositories that are recently updated and actively managed. This increases the likelihood that your contribution will be seen, reviewed, and merged.
Choose Quality Over Quantity ✅
Focus on meaningful contributions that provide real value. Fix bugs, add features, or improve documentation in ways that truly help the project, rather than just seeking to increase your PR count.
Be Transparent and Communicate ✅
If you claim an issue, keep the maintainers informed of your progress. If you can’t complete it, let them know so others can pick it up.
Use Open Source to Learn and Explore New Tech ✅
Leverage your contributions to explore new languages or frameworks, and deepen your understanding of existing ones. This approach makes contributions more enriching for both you and the project.
Ask Questions and Seek Help If Needed ✅
If you’re stuck, don’t hesitate to ask maintainers or the community for help. Clear communication shows dedication and fosters collaboration.

Don’ts ❌

Avoid Low-Effort Contributions ❌
Avoid submitting minor or trivial changes, like typo fixes or formatting updates, unless they are genuinely helpful. Contributions should add value to the project and not clutter the maintainers’ workload.
Don’t Claim Issues You Aren't Committed To Completing ❌

Claiming issues without follow-through disrupts progress and blocks other contributors who may be eager to work on them.
Refrain from Spammy or Artificial Contributions ❌

Resist the temptation to create pull requests on fake or spam repositories just to earn a badge. These actions dilute the purpose of Hacktoberfest and harm the integrity of open source.
Don’t Submit Changes You Haven’t Tested ❌

Testing your code before submission ensures that you’re not introducing bugs or issues into the project. Untested code is a disservice to the maintainers and fellow contributors.
Avoid Relying Entirely on AI Tools for Contributions ❌

While tools like ChatGPT can help with generating documentation or code snippets, ensure that your contributions go beyond automated responses. Genuine contributions involve critical thinking and adaptation specific to the project’s needs.

So What's the point of this?

Hacktoberfest is an opportunity for all of us to learn something new, build and contribute to software all across the world. Who knows what doors it may open for you? Please don't waste it with spam contributions just for the sake of a badge you get to show on LinkedIn.

Even more so, don't waste the contributors time with spam PR's and low effort contributions. Contributing to open source is about more than just adding to your GitHub profile. It’s an opportunity to grow.