Forem: sekka1

Kubernetes Troubleshooting Walkthrough - Pending Pods

sekka1 — Tue, 24 Dec 2019 03:09:09 +0000

Introduction: troubleshooting pending pods

You got your deployment, statefulset, or somehow turned on a pod on the Kubernetes
cluster and it is in a pending state. What can you do now and how do you troubleshoot
it to see what the problem is?

$ kubectl get pods
NAME                                                   READY   STATUS             RESTARTS   AGE
echoserver-657f6fb8f5-wmgj5        0/1     Pending            0          1d

There can be various reasons why your pod is in a pending state. We'll go through them one-by-one and how to
determine what the error messages are telling you.

With any of these errors, step one is to describe the pod:

$ kubectl describe pod echoserver-657f6fb8f5-wmgj5

This will give you additional information. The describe output can be long but look
at the Events section first.

Troubleshooting Reason #1: Not enough CPU

kubectl describe pod echoserver-657f6fb8f5-wmgj5
...
...
Events:
  Type     Reason            Age               From               Message
  ----     ------            ----              ----               -------
  Warning  FailedScheduling  2s (x6 over 11s)  default-scheduler  0/4 nodes are available: 4 Insufficient cpu.

To expand on this line. Kubernetes FailedScheduling of this pod. There are 0 out of 4 nodes
in the cluster that did not have sufficient CPU to allocate to this pod.

This could mean:

You have requested more CPU than any of the nodes has. For example, each node in the cluster has 2 CPU cores and you request 4 CPU cores. This would mean that even if you turned on more nodes in your cluster, Kubernetes will still not be able to schedule it out anywhere.
There is no more capacity in the cluster per the CPU cores you have requested. If it is not the first case, then this would mean that if you had 4 nodes in the cluster and each node has 1 CPU, all of those CPUs has already been requested and allocated to other pods. In this case, you can turn on more nodes in the cluster and your pod will schedule out.

You can check the total number of node via:

$ kubectl get nodes
NAME                             STATUS   ROLES    AGE   VERSION
gke-gar-3-pool-1-9781becc-bdb3   Ready    <none>   12h   v1.11.5-gke.5
gke-gar-3-pool-1-9781becc-d0m6   Ready    <none>   3d    v1.11.5-gke.5
gke-gar-3-pool-1-9781becc-gc8h   Ready    <none>   4h    v1.11.5-gke.5
gke-gar-3-pool-1-9781becc-zj3w   Ready    <none>   20h   v1.11.5-gke.5

Describing a node will give you more details about the capacity of the node:

$ kubectl describe node gke-gar-3-pool-1-9781becc-bdb3
Name:               gke-gar-3-pool-1-9781becc-bdb3
...
...
Allocatable:
 cpu:                940m
 ephemeral-storage:  4278888833
 hugepages-2Mi:      0
 memory:             2702164Ki
 pods:               110
...
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests         Limits
  --------  --------         ------
  cpu       908m (96%)       2408m (256%)
  memory    1227352Ki (45%)  3172952Ki (117%)
...
...

This will tell you how much this node's CPU/memory has been requested. The Request
can never go over 100% but the Limits can. We are interested in the Request
column. For example, this output is telling us that it is at 96% of the max cpu
that is allocatable. This means that we have 4% more we can request. Looking at
the Allocatable cpu section(940m) and the current Request cpu (908m), this means we have (940m - 908m)
32m worth of CPU that we can still request.

Looking back at our describe pod output:

Limits:
  cpu:     16
  memory:  128Mi
Requests:
  cpu:        16
  memory:     64Mi

We can see that we have requested 16 CPU. What happened to the m and why is it 16? This
deserves a little bit of explanation to understand this. CPU request/limits are in the
units of CPU cores. For 1 CPU core it is either 1 or 1000m. This means you can ask for
half a core by donoting 500m.

For this example, we have requested a very high CPU core request at 16 cores. From our
describe node output this node only has 940m it can allocate out which is under one
core which means it will never be able to schedule this pod out on this node type. It
just doesnt have enough CPU cores on it.

On the flip side, even if we requested something reasonable like 1 core, it still wouldn't
be able to schedule it out. We would have to request (per our calculation above) 32m of
CPU.

Troubleshooting Reason #2: Not enough memory

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  2m6s (x25 over 2m54s)  default-scheduler  0/4 nodes are available: 4 Insufficient cpu, 4 Insufficient memory.

We would go through about the same troubleshooting workflow as the CPU above.

The two problems are the same. Either we have requested way too much memory or our nodes just don't
have the memory we are requesting.

We would look at our nodes and see what available memory they have:

$ kubectl describe node gke-gar-3-pool-1-9781becc-bdb3
Name:               gke-gar-3-pool-1-9781becc-bdb3
...
...
Allocatable:
 cpu:                940m
 ephemeral-storage:  4278888833
 hugepages-2Mi:      0
 memory:             2702164Ki
 pods:               110
...
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests         Limits
  --------  --------         ------
  cpu       908m (96%)       2408m (256%)
  memory    1227352Ki (45%)  3172952Ki (117%)
...
...

This node has 1227352Ki memory free. About 1.2 GB.

Now we look at the describe pod output to see how much we have requested:

Limits:
  cpu:     100m
  memory:  125Gi
Requests:
  cpu:        100m
  memory:     64000Mi

We did request a lot of memory for this example; 64GB. Same thing as the CPU, none
of our nodes has this much memory. We either lower the memory request or change
the instance type to have sufficient memory.

Troubleshooting Reason #3: Not enough CPU and memory

Events:
  Type     Reason            Age                     From               Message
  ----     ------            ----                    ----               -------
  Warning  FailedScheduling  2m30s (x25 over 3m18s)  default-scheduler  0/4 nodes are available: 4 Insufficient cpu, 4 Insufficient memory.

This is a combination on both of the above. The event is telling us that there are
not enough CPU and memory to fulfill this request. We will have to run through
the above two troubleshooting workflows and determine what we want to do for both
the CPU and memory. You can alternatively just look at one (CPU or memory), fix that
problem and then look at what Kubernetes is telling you at that point and continue from there.

Kubernetes Troubleshooting Walkthrough - imagepullbackoff

sekka1 — Tue, 24 Dec 2019 02:58:47 +0000

Introduction: troubleshooting the Kubernetes error, imagepullbackoff

I am writing a series of blog posts about troubleshooting Kubernetes. One of the reasons why Kubernetes is so complex is because troubleshooting what went wrong requires many levels of information gathering. It's like trying to find the other end of a string in a tangled ball of strings. In this post, I am going to walk you through troubleshooting the state, imagepullbackoff.

You got your deployment, statefulset, or somehow turned on a pod on the Kubernetes
cluster and it is in a imagepullbackoff state. What can you do now and how do you troubleshoot
it to see what the problem is?

$ kubectl get pods
NAME                                                   READY   STATUS             RESTARTS   AGE
invalid-container-5896955f9f-cg9jg                     1/2     ImagePullBackOff   0          21h

There can be various reasons on why it is in a imagepullbackoff state. First, let's figure out what error message you have and what it's telling you with describe.

$ kubectl describe pod invalid-container-5896955f9f-cg9jg

This will give you additional information. The describe output can be long but look
at the Events section first.

Troubleshooting: Invalid container image

$ kubectl describe pod invalid-container-5896955f9f-cg9jg
...
...
Containers:
  my-container:
    Container ID:   
    Image:          foobartest4
...
...
Events:
  Type     Reason     Age                 From                                     Message
  ----     ------     ----                ----                                     -------
  Normal   Scheduled  115s                default-scheduler                        Successfully assigned dev-k8sbot-test-pods/invalid-container-5896955f9f-r6sgz to gke-gar-3-pool-1-9781becc-gc8h
  Normal   Pulling    113s                kubelet, gke-gar-3-pool-1-9781becc-gc8h  pulling image "gcr.io/google_containers/echoserver:1.0"
  Normal   Pulled     84s                 kubelet, gke-gar-3-pool-1-9781becc-gc8h  Successfully pulled image "gcr.io/google_containers/echoserver:1.0"
  Normal   Created    84s                 kubelet, gke-gar-3-pool-1-9781becc-gc8h  Created container
  Normal   Started    83s                 kubelet, gke-gar-3-pool-1-9781becc-gc8h  Started container
  Normal   BackOff    27s (x4 over 82s)   kubelet, gke-gar-3-pool-1-9781becc-gc8h  Back-off pulling image "foobartest4"
  Warning  Failed     27s (x4 over 82s)   kubelet, gke-gar-3-pool-1-9781becc-gc8h  Error: ImagePullBackOff
  Normal   Pulling    13s (x4 over 114s)  kubelet, gke-gar-3-pool-1-9781becc-gc8h  pulling image "foobartest4"
  Warning  Failed     12s (x4 over 113s)  kubelet, gke-gar-3-pool-1-9781becc-gc8h  Failed to pull image "foobartest4": rpc error: code = Unknown desc = Error response from daemon: repository foobartest4 not found: does not exist or no pull access
  Warning  Failed     12s (x4 over 113s)  kubelet, gke-gar-3-pool-1-9781becc-gc8h  Error: ErrImagePull

There is a long list of events but only a few with the Reason of Failed.

Warning  Failed     27s (x4 over 82s)   kubelet, gke-gar-3-pool-1-9781becc-gc8h  Error: ImagePullBackOff
Warning  Failed     12s (x4 over 113s)  kubelet, gke-gar-3-pool-1-9781becc-gc8h  Failed to pull image "foobartest4": rpc error: code = Unknown desc = Error response from daemon: repository foobartest4 not found: does not exist or no pull access
Warning  Failed     12s (x4 over 113s)  kubelet, gke-gar-3-pool-1-9781becc-gc8h  Error: ErrImagePull

This gives us a really good indication of what the problem is:

Error response from daemon: repository foobartest4 not found: does not exist or no pull access

From here, we either have a non-existent container registry name or we dont have access to it.
Usually a system will not tell you if an item exist or not if you don't have access to it. This
would allow someone to glean more information than they have access to. This is why the error
message can mean multiple things.

As a user you should at this point take a look at the image name and make sure you have the
correct name. If you do, then you should make sure that this container registry for this
image does not require authentication. As a test you can try to pull the same imae from yor laptop
to see if it works locally for you.

Troubleshooting: Invalid container image tag

Another variation to this is if the container tag does not exist:

$ kubectl describe pod invalid-container-5896955f9f-cg9jg
...
...
Containers:
  my-container:
    Container ID:   
    Image:          redis:foobar
...
...
Events:
  Type     Reason     Age                  From                                     Message
  ----     ------     ----                 ----                                     -------
  Normal   Scheduled  12m                  default-scheduler                        Successfully assigned dev-k8sbot-test-pods/invalid-container-tag-85d478dfbd-hddzg to gke-gar-3-pool-1-9781becc-bdb3
  Normal   Pulling    12m                  kubelet, gke-gar-3-pool-1-9781becc-bdb3  pulling image "gcr.io/google_containers/echoserver:1.0"
  Normal   Started    11m                  kubelet, gke-gar-3-pool-1-9781becc-bdb3  Started container
  Normal   Pulled     11m                  kubelet, gke-gar-3-pool-1-9781becc-bdb3  Successfully pulled image "gcr.io/google_containers/echoserver:1.0"
  Normal   Created    11m                  kubelet, gke-gar-3-pool-1-9781becc-bdb3  Created container
  Normal   BackOff    10m (x4 over 11m)    kubelet, gke-gar-3-pool-1-9781becc-bdb3  Back-off pulling image "redis:foobar"
  Normal   Pulling    10m (x4 over 12m)    kubelet, gke-gar-3-pool-1-9781becc-bdb3  pulling image "redis:foobar"
  Warning  Failed     10m (x4 over 12m)    kubelet, gke-gar-3-pool-1-9781becc-bdb3  Error: ErrImagePull
  Warning  Failed     10m (x4 over 12m)    kubelet, gke-gar-3-pool-1-9781becc-bdb3  Failed to pull image "redis:foobar": rpc error: code = Unknown desc = Error response from daemon: manifest for redis:foobar not found
  Warning  Failed     2m1s (x40 over 11m)  kubelet, gke-gar-3-pool-1-9781becc-bdb3  Error: ImagePullBackOff

This is very similar to the previous error but there is a slight difference that can tell us
that it is the image tag. Once again pulling out the pertinent events:

Warning  Failed     10m (x4 over 12m)    kubelet, gke-gar-3-pool-1-9781becc-bdb3  Failed to pull image "redis:foobar": rpc error: code = Unknown desc = Error response from daemon: manifest for redis:foobar not found

The previous error said the repository was not found and this one does not. It tells
you the manifest for redis:foobar not found. This is a very good indication that the
registry redis exist but it didn't find the tag foobar.

You can test and confirm this by trying to pull this image locally on your laptop:

$ docker pull redis:foobar
Error response from daemon: manifest for redis:foobar not found

We receive the save message. If we try a valid tag:

$ docker pull redis:latest
latest: Pulling from library/redis
6ae821421a7d: Already exists
e3717477b42d: Pull complete
8e70bf6cc2e6: Pull complete
0f84ab76ce60: Pull complete
0903bdecada2: Pull complete
492876061fbd: Pull complete
Digest: sha256:dd5b84ce536dffdcab79024f4df5485d010affa09e6c399b215e199a0dca38c4
Status: Downloaded newer image for redis:latest

We are able to successfully pull this image.

This will help us determine what are the valid tags. Or if your registry has a web
GUI, you can go to that also to see what the valid tags are.

Troubleshooting: Unable to pull a private image

As we mentioned above for the invalid image name, a private image that you don't
have access to will return the same error messages.

If you did determine your image is private, you have to give the pod a secret that
has the proper authentication to allow it to pull the image. This can be the same
credential that you use locally to allow you to pull the image or another read only
machine credential.

Either way, you need to do at least two things:

Add the credential secret to Kubernetes
Add the reference of the secret to use in your pod definition

kubectl -namespace <YOUR NAMESPACE> \
create secret docker-registry registry-secret \
--docker-server=https://index.docker.io/v1/ \
--docker-username=<THE USERNAME> \
--docker-password=<THE PASSWORD> \
--docker-email=not-needed@example.com

In this case the secret name is: registry-secret

Then add this reference so that your pod knows to use it:

apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: awesomeapps
spec:
  containers:
    - name: foo
      image: janedoe/awesomeapp:v1
  imagePullSecrets:
    - name: registry-secret

More information: https://kubernetes.io/docs/concepts/containers/images/#referring-to-an-imagepullsecrets-on-a-pod