Forem: Aniket More

How to Fix Kubernetes Errors: CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled, and More

Aniket More — Tue, 02 Sep 2025 05:34:22 +0000

A complete step-by-step guide to solving common Kubernetes pod and node issues — from Pending pods to OOMKilled containers.

💥1. CrashLoopBackOff

⚠️Issue:

A pod starts → crashes → Kubernetes restarts it → crashes again. The cycle continues with backoff delays.

🔍Causes:

Application error (bad code, runtime exception, misconfigured env vars).
Port binding failure (app tries to bind to an already-used port).
Dependency not available (DB, API, config file).
Wrong command/entrypoint in container spec.
Insufficient resources (CPU/memory OOMKilled).

🛠️Fixes:

Check logs:

kubectl logs <pod-name> -n <namespace>
Validate command & args in the pod spec.
Ensure required configs/secrets/env vars are mounted correctly.
Verify resource requests/limits aren’t too low.
Fix application bugs and redeploy.
If OOMKilled → increase memory limit.

⛔2. ImagePullBackOff

⚠️Issue:
Pod can’t pull the container image from the registry.

🔍Causes:

Incorrect image name or tag.
Image doesn’t exist in the registry.
Private registry requires credentials.
Network/DNS issues preventing access to registry.

🛠️Fixes:

Double-check image name & tag:
image: myregistry/myimage:1.0.0
If private registry → create secret:
kubectl create secret docker-registry regcred \ --docker-server=<registry-url> \ --docker-username=<username> \ --docker-password=<password> \ --docker-email=<email>
and reference it in your deployment spec.
Ensure cluster nodes have outbound internet/DNS resolution.
Verify registry is reachable.

🚫3. NodeNotReady

⚠️Issue:
Kubernetes marks a node as NotReady, and pods can’t be scheduled there.

🔍Causes:

Kubelet on the node is stopped or unhealthy.
Node lost connection to control plane (network failure).
Node out of resources (CPU, memory, disk pressure).
Cloud VM is shut down or unreachable.
Incorrect firewall/security group settings blocking traffic.

🛠️Fixes:

Check node status
kubectl describe node <node-name>
SSH into node and check kubelet service:
systemctl status kubelet
Verify disk space, CPU, memory.
Restart kubelet or Docker/Containerd if needed.
Ensure node can reach control plane (API server).
In cloud: confirm VM is running & networking is correct.
Drain and remove bad nodes if they can’t be recovered:
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data kubectl delete node <node-name>

⏳1. Pending

⚠️Issue: Pod stuck in Pending state.

🔍Causes:

No nodes meet scheduling requirements (taints, tolerations, affinity, node selectors).
Insufficient cluster resources (CPU, memory).
PVC requested but storage not provisioned.

🛠️Fixes:

Check scheduling:
kubectl describe pod <pod>
Free or add resources / scale cluster.
Ensure PVC has a valid StorageClass.
Adjust affinity/taints/tolerations as needed.

🐳 2. RunContainerError

⚠️Issue: Pod scheduled but container can’t start.

🔍Causes:

Incorrect container command/entrypoint.
Permission issue mounting volumes.
SecurityContext preventing startup.
Missing binary inside container.

🛠️Fixes:

Inspect events:
kubectl describe pod <pod>
Validate command and args in YAML.
Check mounted volumes/permissions.
Test image locally with docker run.

🔑 3. SecretNotFound

⚠️Issue: Pod references a missing secret.

🔍Causes:

Secret not created.
Typo in secret name.
Wrong namespace (secrets are namespace-scoped).

🛠️Fixes:

Verify secret exists:
kubectl get secret -n <namespace>
Correct secret name in Deployment.
Re-create the secret if deleted.

⚙️ 4. ConfigMapNotFound

⚠️Issue: Pod references a missing ConfigMap.

🔍Causes:

ConfigMap not created yet.
Wrong namespace.
Typo in name.

🛠️Fixes:

Check existence:
kubectl get configmap -n <namespace>
Fix YAML reference.
Apply ConfigMap before deploying pods.

💥 5. OOMKilled

⚠️Issue: Pod restarted due to out-of-memory kill.

🔍Causes:

Container exceeded memory limit.
Application memory leak.
Limit too strict.

🛠️Fixes:

Check pod events:
kubectl describe pod <pod>
Increase resources.limits.memory.
Optimize app memory usage.
Enable monitoring to detect leaks.

🌐 6. DNS Resolution Failure

⚠️Issue: Pod can’t resolve service names / external domains.

🔍 Causes:

coredns pods not running/healthy.
Network policies blocking DNS.
Node network misconfiguration.
Wrong /etc/resolv.conf.

🛠️Fixes:

Check CoreDNS:
kubectl get pods -n kube-system -l k8s-app=kube-dns
Restart CoreDNS if needed.
Validate kubectl exec -- nslookup kubernetes.default.
Review NetworkPolicies / firewall.

🖥️ 7. NodeNotReady / Node Lost

⚠️Issue: Node marked NotReady or Unknown.

🔍 Causes:

Kubelet stopped/crashed.
Node lost connectivity to control plane.
Out of resources (disk, memory, CPU).
VM/physical node powered off.

🛠️Fixes:

Check node status:
kubectl describe node <node>
SSH into node → check kubelet:
systemctl status kubelet
Restart kubelet/container runtime.
Ensure node can reach API server.
Drain & replace node if unrecoverable:
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data kubectl delete node <node>

Forget Manual Fixes—GitOps Is the Future of Software Management

Aniket More — Mon, 01 Sep 2025 15:27:25 +0000

Imagine you want to build a house. Instead of telling different people what to do at different times (which can lead to mistakes), you write down exactly what the house should look like – every wall, every window, every pipe – in a detailed blueprint. Then, you hire a builder who uses this blueprint to make sure the house is built perfectly and stays that way.

GitOps is a bit like that blueprint system for computer software and the machines it runs on.

What is GitOps?

GitOps is a way of managing all your computer systems and applications by using a special kind of file storage called Git as the "single source of truth". Think of Git as a super smart notebook where you write down exactly how your system should look and behave.

Instead of manually changing things on your live computer systems, you write down your desired changes in files (often called YAML or JSON files) and store these files in Git.

This idea started gaining popularity around 2017 with a company called Weaveworks, as more businesses needed better ways to handle complex cloud-based systems, especially with new technologies like Kubernetes.

Why GitOps is Super Useful

GitOps was created to solve common problems that happen when managing computer systems, like things getting messed up or not knowing who changed what.

It's useful because it:

Keeps Everything in Sync: It makes sure your live computer system always matches exactly what you wrote down in Git. No more confusion about why things are different in various places.
Fixes Problems: It helps solve issues where system settings drift apart (called "configuration drift") or when it's hard to trace who made a change.
Reduces Mistakes: Because changes are made automatically, there are fewer human errors.
Makes it Repeatable: It ensures that setting up a new system or fixing an old one can be done the same way every time, which is great for growing bigger.

Key Advantages of Using GitOps

Using GitOps brings many benefits, making everything smoother and more reliable:

Clear History and Tracking: Git keeps a perfect record of every change made, showing who did what and when. This is like having a complete history book for your system, which is great for understanding problems or meeting rules. You can even easily go back to an earlier, working version if something breaks.
Consistency Everywhere: It makes sure all your different environments (like your testing area, your development area, and your live system) are set up exactly the same way. This stops things from getting out of sync.
Automatic Updates: Once you write down a change in Git, tools automatically apply those changes to your live system. This saves time and reduces errors.
Teamwork Made Easy: Many people or teams can work on different parts of the system together, using familiar Git methods like "pull requests" (suggesting a change) and "merges" (accepting a change).
Better Security: It makes your system more secure because people don't directly change the live system. All changes have to go through Git, where they can be reviewed before being applied.
Self-Healing Systems: If someone accidentally (or on purpose) makes a change directly to the live system that isn't in Git, GitOps tools can detect this and automatically fix it by making the system match what's in Git again. This makes your system "self-healing".

Argo CD: A Popular GitOps Helper

Argo CD is a very popular, free tool that helps put GitOps into practice, especially for systems running on Kubernetes. Its main job is to make sure your Kubernetes cluster (where your applications run) always looks exactly like what you've described in your Git repository.

How Argo CD Works with GitOps:

Git is the Master Plan: You put all your plans and settings for your applications (like how they should run, what parts they need) into files in your Git repository.
Argo CD is the Watchdog: A small program (called an agent or controller) from Argo CD runs inside your Kubernetes system. It constantly watches your Git repository for any changes.
Syncing Up: When you make a change in Git (like updating an app or adding a new part), Argo CD notices the difference between what's in Git and what's currently running.
Making it Happen: Argo CD then automatically applies those changes to your Kubernetes system to make it match what's in Git. This replaces old-fashioned manual commands with smart automation.

Good Things About Argo CD:

It's designed to automatically keep things up-to-date.
It has an easy-to-use website (UI) where you can see how your applications are doing.
It's secure because it pulls changes from Git rather than letting other systems push changes directly.
It can fix problems itself if changes are made outside of Git.
Challenges with Argo CD:
It can be a bit tricky to learn at first, as you need to understand Kubernetes and GitOps too.
Managing many different applications and their settings can become complicated.
You need to think carefully about how to handle sensitive information (like passwords) and who can access what.

The Future of GitOps

GitOps and tools like Argo CD are becoming the standard way to manage modern software. What's next?

More Use: More and more companies will start using GitOps and Argo CD.
Better Connections: Argo CD will work even better with other tools, like those for continuous building (CI/CD pipelines) and checking how systems are performing.
Stronger Security: We'll see better ways to handle secrets and control who can do what, making systems even more secure.
Smart Automation: Artificial intelligence (AI) might start helping to predict and prevent problems, or even write some of the system settings for us.
Broader Reach: GitOps won't just be for Kubernetes; it will manage more types of computer infrastructure, including systems spread across different clouds.
Simpler for Developers: GitOps is a key part of "Platform Engineering," which aims to give developers easy-to-use tools to deploy their apps without needing to be experts in the underlying computer systems.

Other Important Ideas in GitOps

Tools: Besides Argo CD, other tools like Flux and Jenkins X also help with GitOps.
Configuration as Code: Everything about your system, from its settings to its structure, is written down as code (usually in YAML files) in Git. This means your system's "state" is literally in code.

In short, GitOps makes managing complex computer systems much simpler, more reliable, and easier to track, by putting all the "master plans" in Git and using smart tools to follow those plans automatically.