Diagnostics on Kubernetes: Obtaining a Memory Dump

Damon Morris — Mon, 10 Jan 2022 19:14:38 +0000

If like me you found that there is a slight memory leak in your application running on Kubernetes and you are searching non-stop on how to collect a memory dump, it can be hard to find exactly how to do this on a .NET application running on Linux-based containers.

There are a handful of articles with useful instructions and tips, however none take into account whether your Docker images are hardened. So below, I've broken down the process into four steps which explain just how exactly to do this:

Installing the dotnet-gcdump tool
Running the tool on your application
Collecting the results
Analysing the results

Pre-requisites

Just some important things to note before we get into the good stuff:

Node OS: Linux
Local machine OS: Windows
Alpine Linux-based containers
.NET 6 SDK & Runtime

The steps below are tailored for this particular setup. They may differ in some way depending on what you yourself are running; for example, the dotnet-gcdump tool is only available in .NET Core 3.1 and above, so be sure to find the details of your setup before attempting this.

Installing the dotnet-gcdump tool

Like I mentioned earlier, if your containers are hardened (i.e. your containers are not running as root) you will not be able to directly install this tool onto your running container.

What you need to do instead is add the instructions to your Dockerfile to install the tool onto your container for it then to be available when your container is running:



FROM mcr.microsoft.com/dotnet/sdk:6.0-alpine3.14 as build
...
RUN dotnet tool install --tool-path /tools dotnet-gcdump
...

In the above example; I am referencing the .NET 6 Alpine SDK base image, from which I am running the dotnet tool install command and installing the dotnet-gcdump tool to the /tools folder in my container.

Running the tool

First things first is to check that your tool has successfully installed onto your container. To do this, boot up an interactive terminal session on one of your pods by running the kubectl exec command with the necessary arguments. From here, change to the tools directory and run the ls command to list the files within this folder, and you should see the dotnet-gcdump tool within.

While you are in this folder, you will want to run the dotnet-gcdump ps command to find the process ID of your running application (if you don't already know what it is). Make a note of this as it will be needed for the next step.

Now we have everything we need to run the dotnet-gcdump collect command. You may find that your application does not have write permissions to the /tools folder if you try to run this command without declaring an output file location. If this is the case, declare the output file location to the temporary directory (/tmp in linux) which should allow your application to write to.

You will also need to give your file a name. In the below example, I have called the file memory-dump-test.

The process should take no longer than a few seconds, and when succeeded, you should see the message Finished writing XXXX bytes.

To confirm, change directories to your /tmp file and run the ls command again to find that your .gcdump file has successfully been generated.

Collecting the results

Now you have generated your memory dump, the next step is porting this file over from your pod to your local machine so you can read the results. In my example below, I used the kubectl cp command. Before executing this command, be sure that you have exited from the terminal session in the previous section!

This command can be fiddly due to needing to match the file path syntax with that of the corresponding operating system (Linux uses '/' while Windows uses '\') as you can see in the screenshot.

There is no success message when this command has completed, but you may or may not see a tar message referring to leading '/' from member names, however if you check the output file you declared, you should see that your gcdump file has successfully ported over.

Analysing the results

The resulting gcdump file can be opened in Visual Studio.

This file gives you the count of a specific object, the size in bytes, and inclusive size (which is the size of all objects that are referenced by the type). Visual Studio also gives you the option to select two memory dumps and compare the difference between them.

I won't go into too much detail about these files, as Microsoft themselves have many great, bitesize videos and documents on how to interpret these results, such as this one.

Conclusion

And that's all there is too it! The solution is fairly trivial in retrospect, however trying to piece together the separate steps can be a bit of a nightmare especially when security measures are in place. Hopefully you can get to the bottom of your memory issues in peace!