Forem: Mumtaz Jahan

How I Fixed Jenkins Built-In Node Offline Issue on EC2

Mumtaz Jahan — Mon, 20 Apr 2026 11:08:28 +0000

Problem: Jenkins Built-In Node Showing Offline on EC2

If you have ever set up Jenkins on an AWS EC2 instance and seen your Built-In Node showing offline with builds not running — this post is for you!

Here is exactly what happened, why it happened, and how I fixed it step by step.

Understanding the Warning

When I clicked on the Built-In Node I saw this error:

Disk space is below threshold of 1.00 GiB. Only 951.90 MiB out of 956.65 MiB left on /tmp.

Jenkins monitors your server's system resources constantly. It requires:

Resource	Minimum Required
Free Disk Space	≥ 1 GB
Free Temp Space `/tmp`	≥ 1 GB
Free Swap Space	> 0 B

In my case the checks showed:

Free Disk Space: 22.26 GiB — perfectly fine
Free Temp Space /tmp: 951.90 MiB — below Jenkins threshold
Free Swap Space: 0 B — no swap at all

The /tmp partition was only 956 MiB total — just under Jenkins' 1 GB requirement. So Jenkins automatically took the Built-In Node offline and refused to run any builds.

Fix Option 1 — Increase `/tmp` Size (Best for EC2)

This is the permanent fix. We increase the /tmp mount size by editing the filesystem configuration.

Step 1: Open the fstab file

sudo nano /etc/fstab

Step 2: Add this line at the bottom

tmpfs /tmp tmpfs defaults,size=2G 0 0

Save and exit (Ctrl+X → Y → Enter)

Step 3: Remount /tmp without rebooting

sudo mount -o remount /tmp

Step 4: Verify the new size

df -h /tmp

You should now see:

Filesystem   Size   Used   Avail   Use%   Mounted on
tmpfs        2.0G   4.8M   2.0G    1%     /tmp

/tmp is now 2 GB — well above Jenkins' 1 GB threshold! ✅

Fix Option 2 — Quick Jenkins Restart (Temporary Fix)

If you just need Jenkins back online quickly:

sudo rm -rf /tmp/*
sudo systemctl restart jenkins

This clears temp files and forces Jenkins to recheck the threshold. Sometimes this is enough to bring the node back online temporarily.

Bonus Fix — Add Swap Space (Important for t2.micro)

On t2.micro EC2 instances, swap is 0 B by default. Jenkins warns about this too. Here is how to create a 1 GB swap file:

# Create a 1GB swap file
sudo fallocate -l 1G /swapfile

# Set correct permissions
sudo chmod 600 /swapfile

# Set up swap space
sudo mkswap /swapfile

# Enable the swap
sudo swapon /swapfile

# Verify
free -h

Result

After applying Fix Option 1 and adding swap space the node came back online immediately!

Built-In Node came back Online
Jenkins disk space warning disappeared
Builds started running immediately

Key Takeaways

t2.micro has very limited resources — always check /tmp size when setting up Jenkins on a free tier EC2.

Never ignore Jenkins resource warnings — they directly affect whether your node stays online.

Fix Option 1 is permanent, Fix Option 2 is temporary. Always go with Option 1 for a stable setup.

Always add swap on t2.micro — it prevents a lot of memory-related Jenkins issues down the line.

*Setting up Jenkins on EC2? Drop your questions in the comments *

Git Commands Every DevOps Engineer Must Know

Mumtaz Jahan — Mon, 20 Apr 2026 03:23:56 +0000

Git Commands Every DevOps Engineer Must Know

Git is not just a version control tool — it's your daily survival kit as a DevOps engineer. Whether you're managing pipelines, fixing production issues, or collaborating with teams, these commands will save you every single day.

Let's break it down section by section.

1. Initial Setup — Configure Git & Start Your Project

Before anything else, set up Git properly:

# Set your global username
git config --global user.name "Your Name"

# Set your global email
git config --global user.email "your@email.com"

# Initialize a new Git repository
git init

# Clone an existing repository
git clone <repo-url>

Tip: Start every project the right way — proper config avoids identity issues in commits later.

2. Daily Git Flow — Your Everyday DevOps Cycle

This is the cycle you'll repeat every single day:

# Check the status of your changes
git status

# Stage all changes to be committed
git add .

# Commit your changes with a message
git commit -m "your message here"

# Push changes to remote repository
git push

Tip: Practice it. Automate it. Master it.

3. Branching Strategy — Work Smarter in CI/CD & Teamwork

Branching is essential for clean CI/CD pipelines:

# List all branches in your repository
git branch

# Create a new branch and switch to it
git checkout -b feature-branch

# Switch to an existing branch
git switch branch-name

# Merge changes from another branch
git merge branch-name

Tip: Always keep your main branch clean and stable. Never push untested code directly to main.

4. Sync With Remote — Keep Your Local Repo Up to Date

Always sync before you push:

# Fetch and merge changes from remote to local
git pull

# Fetch changes from remote (without merging)
git fetch

# Show remote repositories and their URLs
git remote -v

Tip: Always git pull before you git push — it helps you avoid conflicts and ensures successful deployments.

5. Debug Like a Pro — Inspect History, Changes & Code

When something breaks in production, these are your best friends:

# Show commit history
git log

# View changes between working tree, staging or commits
git diff

# Show details of a specific commit
git show <commit-id>

# See who changed each line and why
git blame <file>

Tip: git blame helps you find who broke production 👀 — use it wisely!

6. Undo & Fix — Life Saver Commands

Mistakes happen. Fix them like a pro:

# Undo last commit, keep changes staged
git reset --soft HEAD~1

# Undo last commit and discard all changes (Use with caution!)
git reset --hard HEAD~1

# Safely undo changes by creating a new commit (great for shared repos)
git revert <commit-id>

# Temporarily save changes and clean your working directory
git stash

Tip: Use git revert over git reset --hard in shared/production repos — it's safer and keeps history clean.

7. Bonus — Extra Power Commands

Level up your Git game:

# Create a tag for a specific version
git tag -a v1.0 -m "version 1.0"

# Push a specific tag to remote
git push origin v1.0

# Apply a specific commit from another branch
git cherry-pick <commit-id>

# Check repository integrity and find issues
git fsck --full

# Remove untracked files and directories
git clean -fd

Tip: Master these commands and you're ready to tackle any DevOps workflow with confidence!

Quick Reference Summary

Command	What it does
`git status`	Check current changes
`git pull`	Sync from remote
`git stash`	Save work temporarily
`git revert`	Safely undo in shared repos
`git blame`	Find who changed what
`git cherry-pick`	Apply specific commits
`git tag`	Version your releases

Final Thought

Git is the backbone of every DevOps workflow. The engineers who know these commands deeply don't just push code — they ship with confidence.

Save this post for your next deployment!

*Which Git command has saved you the most in production? Drop it in the comments *

DevOps Scenario Interview Question: Deployment Failed in Production

Mumtaz Jahan — Sat, 18 Apr 2026 14:11:02 +0000

Tags: devops kubernetes cicd career

Scenario: Your Deployment Failed in Production. What Steps Will You Take?

This is one of the most common real-world scenario questions asked in DevOps interviews. Interviewers don't want textbook answers — they want to know how you think under pressure.

Here's the complete answer framework.

Answer: Step-by-Step Approach

1. Check CI/CD Pipeline Logs

First thing — don't guess, read the logs.

# For Jenkins
cat /var/log/jenkins/jenkins.log

# For GitHub Actions — check the Actions tab in your repo

# For GitLab CI
gitlab-ci logs

The pipeline log tells you exactly where it broke.

2. Identify the Failed Stage (Build / Test / Deploy)

Every pipeline has stages. Narrow it down:

Build failed? → Dependency issue, Dockerfile error, compilation error
Test failed? → A test caught a regression before it hit production
Deploy failed? → Kubernetes issue, wrong image tag, resource limits, misconfigured secrets

Knowing the stage cuts your debugging time in half.

3. Verify Configuration Changes

Check what changed before the failure:

# Check recent git commits
git log --oneline -10

# Check Kubernetes config changes
kubectl describe deployment my-app

# Check if secrets/configmaps were updated
kubectl get configmap my-app-config -o yaml

Most production failures trace back to a config change someone forgot to mention.

4. Rollback to Previous Stable Version

Don't try to fix forward when production is down. Rollback first, fix later.

# Kubernetes rollback
kubectl rollout undo deployment/my-app

# Verify rollback status
kubectl rollout status deployment/my-app

# Check rollout history
kubectl rollout history deployment/my-app

This restores service immediately while you investigate the root cause safely.

5. Fix the Issue and Redeploy

Once production is stable:

Reproduce the issue in staging
Apply the fix
Test thoroughly
Redeploy with the corrected version

kubectl set image deployment/my-app my-app=my-image:v2.1-fixed
kubectl rollout status deployment/my-app

Pro Tip

Always maintain versioned Docker images — never use latest in production.

# Bad
image: my-app:latest

# Good
image: my-app:v2.0.1

Without versioned images, you can't rollback. Tag every release.

Bonus: What Interviewers Are Really Looking For

They want to see that you: don't panic, prioritize restoring service over finding blame, think in structured steps, and know the actual commands — not just theory.

*Preparing for a DevOps interview? Drop your toughest scenario question in the comments *

How to Fixed a Kubernetes CrashLoopBackOff in Production

Mumtaz Jahan — Fri, 17 Apr 2026 23:55:42 +0000

Tags: kubernetes devops debugging cloud

Problem: Application Was DOWN in Kubernetes

One of the most stressful moments in DevOps — you check your monitoring dashboard and your application is completely DOWN in Kubernetes. No graceful degradation. Just... down.

Here's exactly how I diagnosed and fixed it in under an hour.

Issue Found: Pod Was in CrashLoopBackOff

Running kubectl get pods revealed the culprit immediately:

NAME                        READY   STATUS             RESTARTS   AGE
my-app-7d9f8b6c4-xk2pq     0/1     CrashLoopBackOff   8          20m

CrashLoopBackOff means Kubernetes is repeatedly trying to start your container, it crashes, and Kubernetes backs off with increasing wait times before retrying. Something inside the container was causing it to exit immediately on startup.

Debug Steps

Step 1: Checked Logs (`kubectl logs`)

kubectl logs my-app-7d9f8b6c4-xk2pq --previous

The --previous flag is crucial here — it lets you see logs from the crashed container, not the current (possibly empty) one. The logs showed repeated connection errors on startup.

Step 2: Checked Config

kubectl describe pod my-app-7d9f8b6c4-xk2pq

I inspected the environment variables and ConfigMaps attached to the pod. The describe command is a goldmine — it shows events, resource limits, volume mounts, and more.

Step 3: Found DB Connection Issue

The logs made it clear: the app was trying to connect to the database using an incorrect connection string. The host value in the environment variable was pointing to a stale endpoint. The app would crash immediately on boot since it couldn't reach the DB.

Fix Applied

Corrected Environment Variables

Updated the Kubernetes secret/configmap with the correct database host:

kubectl edit secret my-app-db-secret
# or
kubectl set env deployment/my-app DB_HOST=correct-db-host.internal

Restarted the Deployment

kubectl rollout restart deployment/my-app

Then watched the rollout:

kubectl rollout status deployment/my-app

🎉 Result

 Application UP
 Issue resolved

The pods came up healthy, readiness probes passed, and traffic started flowing again.

Key Takeaways

Always check logs with --previous — the live container may have no logs if it crashes before writing any.

kubectl describe pod is your best friend for seeing the full picture: events, env vars, resource pressure.

CrashLoopBackOff is almost always one of: bad env vars/secrets, missing config, OOM kill, or a bug triggered at startup.

kubectl rollout restart is safer than deleting pods manually — it does a rolling restart with zero downtime.

*Hit a similar issue? Drop your debugging story in the comments *

Forem: Mumtaz Jahan

How I Fixed Jenkins Built-In Node Offline Issue on EC2

Problem: Jenkins Built-In Node Showing Offline on EC2

Understanding the Warning

Fix Option 1 — Increase /tmp Size (Best for EC2)

Fix Option 2 — Quick Jenkins Restart (Temporary Fix)

Bonus Fix — Add Swap Space (Important for t2.micro)

Result

Key Takeaways

Git Commands Every DevOps Engineer Must Know

Git Commands Every DevOps Engineer Must Know

1. Initial Setup — Configure Git & Start Your Project

2. Daily Git Flow — Your Everyday DevOps Cycle

3. Branching Strategy — Work Smarter in CI/CD & Teamwork

4. Sync With Remote — Keep Your Local Repo Up to Date

5. Debug Like a Pro — Inspect History, Changes & Code

6. Undo & Fix — Life Saver Commands

7. Bonus — Extra Power Commands

Quick Reference Summary

Final Thought

DevOps Scenario Interview Question: Deployment Failed in Production

Scenario: Your Deployment Failed in Production. What Steps Will You Take?

Answer: Step-by-Step Approach

1. Check CI/CD Pipeline Logs

2. Identify the Failed Stage (Build / Test / Deploy)

3. Verify Configuration Changes

4. Rollback to Previous Stable Version

5. Fix the Issue and Redeploy

Pro Tip

Bonus: What Interviewers Are Really Looking For

How to Fixed a Kubernetes CrashLoopBackOff in Production

Problem: Application Was DOWN in Kubernetes

Issue Found: Pod Was in CrashLoopBackOff

Debug Steps

Step 1: Checked Logs (kubectl logs)

Step 2: Checked Config

Step 3: Found DB Connection Issue

Fix Applied

Corrected Environment Variables

Restarted the Deployment

🎉 Result

Key Takeaways

Fix Option 1 — Increase `/tmp` Size (Best for EC2)

Step 1: Checked Logs (`kubectl logs`)