Forem: Indy Singh

Optimising Image Processing

Indy Singh — Wed, 11 Jul 2018 11:32:08 +0000

Reducing memory consumption and time taken by 80%

Investigating old code
Streaming is winning
Pool your streams
Goodbye System.Drawing, you will not be missed
In the interest of transparency
TLDR - tell me what to do

Investigating old code

In my previous article, we covered how we found and solved a fixed cost of using PutObject on the AWS S3 .NET client; it was also mentioned that we transform the image before it is uploaded to AWS S3 - this is a process that is executed at least 400,000 times a day. With that in mind I decided it was worth exploring the entire code path. As always you can find all the code talked about here, and we will be using this image in our testing. Let us take a look at the existing implementation. Before we can transform an image we need to get a hold of it:-

private static byte[] GetImageFromUrl(string url)
{
    byte[] data;
    var request = (HttpWebRequest)WebRequest.Create(url);
    request.Timeout = 10000;
    request.ReadWriteTimeout = 10000;

    using (var response = request.GetResponse())
    using (var responseStream = response.GetResponseStream())
    using (var memoryStream = new MemoryStream())
    {
        int count;
        do
        {
            var buf = new byte[1024];
            count = responseStream.Read(buf, 0, 1024);
            memoryStream.Write(buf, 0, count);
        } while (responseStream.CanRead && count > 0);

        data = memoryStream.ToArray();
    }

    return data;
}

Do you see that .ToArray()? It is evil. I do not have the statistics to hand but I know for a fact I have seen images larger than four megabytes come through this code path. That does not bode well - a quick peak at the implementation shows us why:-

public virtual byte[] ToArray()
{
    BCLDebug.Perf(_exposable, "MemoryStream::GetBuffer will let you avoid a copy.");
    byte[] copy = new byte[_length - _origin];
    Buffer.InternalBlockCopy(_buffer, _origin, copy, 0, _length - _origin);
    return copy;
}

The LOH threshold is 85,000 bytes this would mean any images larger than the threshold would go straight onto the LOH - this is a problem, I covered why it is a problem in my previous article. At this point curiosity got the better of me, I really wanted to know the size of images coming into this service.

The service processes hundreds of thousands images per day, all I need is a quick sample and idea of the size and shape of incoming data. We have a job that runs at 14:00 every day to pull the latest images from one of our customers.

Hitting their service manually returns 14,630 images to be processed. Spinning up a quick console app to perform a HEAD request and get the Content-Length header:-

Total Images: 14,630
Images Above LOH Threshold: 14,276
Average Image Size: 253,818 bytes
Largest Image Size: 693,842 bytes
Smallest Image Size: 10,370 bytes
Standard Deviation of Sizes: 101,184

Yikes, that is a huge standard deviation and 97.58% of the images are above the LOH threshold (85,000 bytes). Now that we know the spread of image sizes, we can resume looking at the rest of the current implementation:-

public void Transform(string url)
{
    var bytes = GetImageFromUrl(url);

    if (CanCreateImageFrom(bytes))
    {
        using (var stream = new MemoryStream(bytes))
        using (var originalImage = Image.FromStream(stream))
        using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
        using (var graphics = Graphics.FromImage(scaledImage))
        {
            ImageHelper.TransformImage(graphics, scaledImage, originalImage);

            // upload scaledImage to AWS S3 in production, in the test harness write to disk
            using (var fileStream = File.Create(@"..\..\v1.jpg"))
            {
                scaledImage.Save(fileStream, ImageFormat.Jpeg);
            }
        }
    }
}

private static bool CanCreateImageFrom(byte[] bytes)
{
    try
    {
        using (var stream = new MemoryStream(bytes))
        {
            Image.FromStream(stream);
        }
    }
    catch (ArgumentException)
    {
        return false;
    }

    return true;
}

I have suspicions about the two MemoryStream's knocking about but dotTrace can confirm or deny my suspicions. Running V1 with the sample image one hundred times:-

	V1
Took (ms)	10,297
Allocated (kb)	851,894
Peak Working Set (kb)	96,276
Gen 0 collections	184
Gen 1 collections	101
Gen 2 collections	101
	-
dotTrace Total RAM (MB)	901
dotTrace SOH (MB)	410
dotTrace LOH (MB)	491

Using dotTrace we can see where the biggest costs are:-

Now that we have established a baseline - lets get cracking!

Streaming is winning

If we inline the GetImageFromUrl and remove the CanCreateImageFrom - which is not needed because we check the Content-Type earlier in the code path, we can directly operate on the incoming stream.

public class ImageTransformerV2 : IImageTransformer
{
    public void Transform(string url)
    {
        var request = WebRequest.CreateHttp(url);
        request.Timeout = 10000;
        request.ReadWriteTimeout = 10000;

        using (var response = request.GetResponse())
        using (var responseStream = response.GetResponseStream())
        using (var originalImage = Image.FromStream(responseStream))
        using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
        using (var graphics = Graphics.FromImage(scaledImage))
        {
            ImageHelper.TransformImage(graphics, scaledImage, originalImage);

            // upload scaledImage to AWS S3 in production, in the test harness write to disk
            using (var fileStream = File.Create(@"..\..\v2.jpg"))
            {
                scaledImage.Save(fileStream, ImageFormat.Jpeg);
            }
        }
    }
}

Stats for V2:-

	V1	V2	%
Took (ms)	10,297	7,844	-23.82%
Allocated (kb)	851,894	527,246	-38.10%
Peak Working Set (kb)	96,276	69,436	-27.87%
Gen 0 collections	184	100	-45.65%
Gen 1 collections	101	100	-00.99%
Gen 2 collections	101	100	-00.99%
	-
dotTrace Total RAM (MB)	901	550	-38.95%
dotTrace SOH (MB)	410	162	-60.48%
dotTrace LOH (MB)	491	388	-20.97%

Awesome, a few minor tweaks and all the metrics have dropped across the board.

Pool your streams

Using dotTrace again, we can see the next biggest cost:-

Something is happening inside of the constructor of GPStream that is costing us dearly. Luckily dotTrace can show us the decompiled source which saves us a trip to Reference Sources:-

internal GPStream(Stream stream)
{
    if (!stream.CanSeek)
    {
        byte[] buffer = new byte[256];
        int offset = 0;
        int num;

        do
        {
            if (buffer.Length < offset + 256)
            {
                byte[] numArray = new byte[buffer.Length * 2];
                Array.Copy((Array) buffer, (Array) numArray, buffer.Length);
                buffer = numArray;
            }
            num = stream.Read(buffer, offset, 256);
            offset += num;
        } while (num != 0);

        this.dataStream = (Stream) new MemoryStream(buffer);
    }
    else
        this.dataStream = stream;
}

You know when you ask yourself a question out loud and you can answer it straight away?

"What?! I can't seek on the incoming HTTP stream?"

No, you can't. Makes total sense when you think about it and because the incoming Stream is not seekable GPStream has to take a copy of it.

Okay, first thing - can we move the cost from framework code to our code? It is not pretty but something like this works:-

public void Transform(string url)
{
    var request = WebRequest.CreateHttp(url);
    request.Timeout = 10000;
    request.ReadWriteTimeout = 10000;

    var memoryStream = new MemoryStream();

    using (var response = request.GetResponse())
    using (var responseStream = response.GetResponseStream())
    {
        responseStream.CopyTo(memoryStream);
    }

    using (var originalImage = Image.FromStream(memoryStream))
    using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
    using (var graphics = Graphics.FromImage(scaledImage))
    {
        ImageHelper.TransformImage(graphics, scaledImage, originalImage);

        // upload scaledImage to AWS S3 in production, in the test harness write to disk

        using (var fileStream = File.Create(@"..\..\v2.jpg"))
        {
            scaledImage.Save(fileStream, ImageFormat.Jpeg);
        }
    }
}

dotTrace shows us that we have indeed moved the cost from framework code into our code:-

Now we could use our old friend System.Buffers and just rent a byte[] then pass that to the constructor of the MemoryStream. That would work, except two things. Firstly, Content-Length is not guaranteed to be set. Secondly, a few weeks earlier I was poking around the .NET driver for Cassandra when I came across Microsoft.IO.RecyclableMemoryStream and it felt like it was exactly what I needed here. If you want to learn more about RecyclableMemoryStream Ben Watson has a great post on what it is and its various use cases. Jumping straight into V3:-

public class ImageTransformerV3 : IImageTransformer
{
    private readonly RecyclableMemoryStreamManager _streamManager;

    public ImageTransformerV3()
    {
        _streamManager = new RecyclableMemoryStreamManager();
    }

    public void Transform(string url)
    {
        var request = WebRequest.CreateHttp(url);
        request.Timeout = 10000;
        request.ReadWriteTimeout = 10000;
        request.AllowReadStreamBuffering = false;
        request.AllowWriteStreamBuffering = false;

        MemoryStream borrowedStream;

        using (var response = request.GetResponse())
        {
            if (response.ContentLength == -1) // Means that content length is NOT sent back by the third party server
            {
                borrowedStream = _streamManager.GetStream(url); // then we let the stream manager had this 
            }
            else
            {
                borrowedStream = _streamManager.GetStream(url, (int)response.ContentLength); // otherwise we borrow a stream with the exact size
            }

            int bufferSize;

            if (response.ContentLength == -1 || response.ContentLength > 81920)
            {
                bufferSize = 81920;
            }
            else
            {
                bufferSize = (int) response.ContentLength;
            }

            // close the http response stream asap, we only need the contents, we don't need to keep it open
            using (var responseStream = response.GetResponseStream())
            {
                responseStream.CopyTo(borrowedStream, bufferSize);
            }
        }

        using (borrowedStream)
        using (var originalImage = Image.FromStream(borrowedStream))
        using (var scaledImage = ImageHelper.Scale(originalImage, 320, 240))
        using (var graphics = Graphics.FromImage(scaledImage))
        {
            ImageHelper.TransformImage(graphics, scaledImage, originalImage);
            // upload scaledImage to AWS S3 in production, in the test harness write to disk

            using (var fileStream = File.Create(@"..\..\v3.jpg"))
            {
                scaledImage.Save(fileStream, ImageFormat.Jpeg);
            }
        }
    }
}

One of my favourite things about RecyclableMemoryStream is that it is a drop-in replacement for MemoryStream. The ArrayPool from System.Buffers requires you to Rent and then Return. Whereas RecyclableMemoryStream handles everything for you as it implements IDisposable. Anyway, enough of my admiration for RecyclableMemoryStream; stats for V3:-

	V1	V2	V3	% (V3 vs. V1)
Took (ms)	10,297	7,844	7,688	-25.33%
Allocated (kb)	851,894	527,246	125,739	-85.24%
Peak Working Set (kb)	96,276	69,436	71,140	-26.10%
Gen 0 collections	184	100	29	-84.23%
Gen 1 collections	101	100	2	-98.01%
Gen 2 collections	101	100	1	-99.00%
-
dotTrace Total RAM (MB)	901	550	152	-83.12%
dotTrace SOH (MB)	410	162	150	-63.41%
dotTrace LOH (MB)	491	388	1.6	-99.67%

That is an incredible improvement from V1. This is what V3 looks like in dotTrace:-

Goodbye System.Drawing, you will not be missed

My knowledge on System.Drawing is pretty sketchy but an afternoon reading about it leads me to the conclusion that if you can avoid using System.Drawing then you are better off. Whilst searching for an alternative to System.Drawing, I came across this article written by Omar Shahine.
Huh, I never considered the overloads. Takes two seconds to try this so we might as well; v3 with useEmbeddedColorManagement disabled and validateImageData turned off these are the stats:-

	V1	V2	V3	% (V3 vs. V1)
Took (ms)	10,297	7,844	7,563	-26.55%
Allocated (kb)	851,894	527,246	128,166	-84.95%
Peak Working Set (kb)	96,276	69,436	53,688	-44.23%
Gen 0 collections	184	100	30	-83.69%
Gen 1 collections	101	100	2	-98.01%
Gen 2 collections	101	100	1	-99.00%
-
dotTrace Total RAM (MB)	901	550	154	-82.90%
dotTrace SOH (MB)	410	162	153	-62.68%
dotTrace LOH (MB)	491	388	1.6	-99.67%

A minor increase in some areas but a noticeable drop in Peak Working Set - great! A slightly more recent article by Bertrand Le Roy goes through the various alternatives to System.Drawing. Thankfully, there is a nice little chart at the bottom that shows performance in the context of time taken.

According to that article PhotoSauce.MagicScaler is pretty magic - word of warning this library is Windows only. I wonder how magic it really is? Spinning V4 up:-

public class ImageTransformerV4 : IImageTransformer
{
    public void Transform(string url)
    {
        // truncated for brevity 

        MagicImageProcessor.EnableSimd = false;
        MagicImageProcessor.EnablePlanarPipeline = true;

        using (borrowedStream)
        {
            // upload scaledImage to AWS S3 in production, in the test harness write to disk

            using (var fileStream = File.Create(@"..\..\v4.jpg"))
            {
                MagicImageProcessor.ProcessImage(borrowedStream, fileStream, new ProcessImageSettings()
                {
                    Width = 320,
                    Height = 240,
                    ResizeMode = CropScaleMode.Max,
                    SaveFormat = FileFormat.Jpeg,
                    JpegQuality = 70,
                    HybridMode = HybridScaleMode.Turbo
                });
            }
        }
    }
}

And the stats:-

	V1	V2	V3	V4	% (V4 vs. V1)
Took (ms)	10,297	7,844	7,563	1,672	-83.76%
Allocated (kb)	851,894	527,246	128,166	135,876	-84.05%
Peak Working Set (kb)	96,276	69,436	53,688	35,596	-63.02%
Gen 0 collections	184	100	30	32	-82.60%
Gen 1 collections	101	100	2	2	-98.01%
Gen 2 collections	101	100	1	1	-99.00%
-
dotTrace Total RAM (MB)	901	550	154	165	-81.68%
dotTrace SOH (MB)	410	162	153	162	-60.48%
dotTrace LOH (MB)	491	388	1.6	1.6	-99.67%

An insane drop in time taken and a healthy drop in Peak Working Set!

That is magic.

In the interest of transparency

Pun not intended - just before this article was about to be published I saw this interesting tweet:-

// Detect dark theme var iframe = document.getElementById('tweet-1012632551486205952-400'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1012632551486205952&theme=dark" }

Naturally I was interested, spinning up V5 gave me similar results to V4 except a noticeable increase in time taken, peak working set, and LOH allocations. After a conversation on their Gitter I learnt that they are:-

Still working on improving their JPEG decoder - this is relevant because we process other image formats too
Aware that their resizing algorithm is not as optimal as it could be - this is relevant because the main part of the transformation is resizing
Happy to be featured in this article (which says a lot)

TLDR - tell me what to do

If your image transformation process is mostly resizing and you are hosted on Windows then look into using PhotoSauce.MagicScaler with Microsoft.IO.RecyclableMemoryStream you will see a sixty to ninety nine percent reduction in various performance related metrics!

Find me Twitter, LinkedIn, or GitHub.

AWS S3 .NET Client High Memory Usage

Indy Singh — Thu, 21 Jun 2018 18:31:36 +0000

Reducing AWS S3 .NET client LOH allocations by 98%

Problem discovery
Why is it a problem?
Introducing the best magic number - 81,920
Idle hands
Just one more thing
TLDR - Give me the good stuff
Footnotes

Problem discovery

One of the things we do at Codeweavers is help people find their next vehicle. That usually involves customers seeing what vehicle they are buying - I mean, would you buy a car without seeing what it looks like? The application that holds this responsibility is the worst offender for obscene amounts of allocations, time spent in GC, and generally eating RAM like the Cookie Monster eats well...cookies.

Every now and then we like to take a memory dump of this application from our production environment. We have done this enough times that we have automated the most common diagnostics steps we take and bundled them into a little tool¹ called ADA (Automated Dump Analysis). If you are interested you can find the tool here and all the code talked about in this article here.

One of the analysers that we run is to dump all the byte[] arrays found on the Large Object Heap (LOH). After running that analyser against our eight gigabyte memory dump, we found several hundred byte[] arrays with a length of 131,096 or 131,186. Well that is pretty odd. Opening some of the files in Notepad++ just presented us with lots of random characters.

Throwing the scientific method out of the window for a second, I decided to mass rename all the dumped byte[] arrays to *.jpg - hey presto some of the files were now displaying thumbnails! On closer inspection, around 50% of the files were images. The other 50% failed to open as an image at all. Opening a handful of the non-image files in Notepad++ showed that they all had a line similar to this right at the beginning of the file:-

0;chunk-signature=48ebf1394fcc452801d4ccebf0598177c7b31876e3fbcb7f6156213f931b261d

Okay, this is beginning to make a little more sense. The byte[] arrays that have a length of 131,096 are pure images. The byte[] arrays that are not images have a length of 131,186 and have a chunk-signature line before the rest of the contents. I would guess the signature is a SHA256 hash of the contents.

Before we go any further, it is worth establishing how busy this application is with image processing. All of our image processing is distributed across our farm using AWS SNS and SQS. Using CloudWatch Metrics we can see that easily:-

Okay, so fairly busy. It is worth noting that before any performance centric work is carried out, always establish how often the code is hit and the current costs. If a code path has a high cost (e.g. takes twenty seconds) but is only hit once a day, then it is not worth investigating. However, if the same code path is hit a lot (e.g. a million times a day) then it is definitely worth investigating.

At this point I had two culprits in mind. We have already established the application in question does a lot of image processing. But there are a few moving parts and two ways of kicking off the image processing:-

Images are pushed to us
We pull images from a SFTP

After that we transform the image and then upload it to AWS S3. At this stage I was leaning towards the SFTP, as it probably needed to verify each chunk it received from the server. But following my hunches has lead me on a wild goose chase before so ignoring my hunch I plugged chunk-signature into Google and smashed enter. Google pointed to AWS S3 as the culprit. But that is just theory, we need to prove that.

public class UploaderV1 : IUploader
{
    private readonly AWSCredentials _credentials;
    private readonly RegionEndpoint _region;
    private readonly string _bucketName;

    public UploaderV1(AWSCredentials credentials, RegionEndpoint region, string bucketName)
    {
        _credentials = credentials;
        _region = region;
        _bucketName = bucketName;
    }

    public void Upload(Stream objectToUpload, int iteration)
    {
        try
        {
            using (var client = new AmazonS3Client(_credentials, _region))
            {
                var putObjectRequest = new PutObjectRequest
                {
                    BucketName = _bucketName,
                    Timeout = TimeSpan.FromSeconds(5),
                    InputStream = objectToUpload,
                    Key = "v1/" + iteration + ".png",
                    CannedACL = S3CannedACL.PublicRead,
                    StorageClass = S3StorageClass.Standard,
                    ServerSideEncryptionMethod = ServerSideEncryptionMethod.None,
                };

                client.PutObject(putObjectRequest);
            }
        }
        catch (Exception exception)
        {
            Console.WriteLine(exception);
        }
    }
}

If we upload the same image ten times and use dotTrace to view the LOH we see an interesting pattern:-

It looks like every time we call PutObject on the AWS S3 .NET client there is a fixed cost of 0.3 MB. This is a problem because it means every time you use PutObject, you are paying a high cost of 0.3 MB per upload. Just to make sure; what happens if we increase the number of times we upload from ten times to one hundred times?

Yes, we can definitely say that for every invocation of PutObject a costly allocation of 0.3 MB is made. Going one step further and dumping the process using ProcDump:-

procdump64.exe -ma -64 AWS-S3.exe

Running the dump file through ADA we see the exact same characteristics of there being two groups of byte[] arrays; 50% have a length 131,096 and the other 50% have a length of 131,186. Half the files are images when renamed, and half the files have the chunk-signature starting line. At this point we are certain that the AWS S3 .NET client is allocating byte[] arrays directly onto the LOH; and that is a problem.

Why is it a problem?

The LOH is a region of memory that is collected but never compacted - though as of .NET v4.5.1 compaction is now possible - word of warning compaction of the LOH is expensive; around 2.3 milliseconds per megabyte. A good rule of thumb is that short-lived objects should never make it onto the LOH.

Objects that are equal to or greater than 85,000 bytes go straight onto the LOH. The LOH operates very differently from other regions of memory. Other regions of memory get collected and compacted regularly meaning that you can just add new objects to the end after the garbage collector runs. Whereas the LOH tries to fit newly allocated objects in free space left after dead objects are discarded. This works fine if the newly allocated object is the exact same size or smaller as the free space. If a space can not be found then the LOH has to grow to accommodate that object.

It helps to think of it like a bookshelf; in other regions of memory, books that are no longer used are simply thrown away and the remaining books are pushed together and any new books go at the end of the book shelf.

Within the LOH that is not possible, instead books (objects) are thrown away, and the number of pages that used to be in that space (bytes) is recorded and the next time a book gets allocated to the that shelf (the LOH) it attempts to find an empty space that can hold that many pages (bytes). If the shelf can not accommodate the newly allocated book (object) then the shelf must be extended to hold that new book (object).

The garbage collector will collect dead objects from the LOH, and in the mean time new objects are being allocated to the LOH. This can lead to a situation over a lifetime of a long running application where the LOH size has grown to a few gigabytes (because new objects did not fit into existing empty space) but actually only contains a few alive objects. This is known as LOH fragmentation. We were extremely lucky in this situation as the byte[] arrays that made it onto the LOH had two sizes; 131,186 and 131,096. This means that as old objects of either size died and were collected, newly allocated objects were just the right size to slot right into the empty space.

Okay, back to the fun stuff.

Introducing the best magic number — 81,920

Thanks to dotTrace we were able to establish exactly what was causing the LOH fragmentation. It also showed us that the fixed cost of 0.3 MB per invocation of PutObject happened inside of the constructor for ChunkedUploadWrapperStream:-

A quick visit to that file in aws-sdk-net repository. Shows that two byte[] arrays are created with a length of at least 131,072:-

public class ChunkedUploadWrapperStream : WrapperStream
{
    public static readonly int DefaultChunkSize = 128*1024;
    private byte[] _inputBuffer;
    private readonly byte[] _outputBuffer;

    // truncated for brevity

    internal ChunkedUploadWrapperStream(Stream stream, int wrappedStreamBufferSize, AWS4SigningResult headerSigningResult) : base(stream)
    {
        // truncated for brevity
        _inputBuffer = new byte[DefaultChunkSize];
        _outputBuffer = new byte[CalculateChunkHeaderLength(DefaultChunkSize)];
    }

    // truncated for brevity
}

This is exactly why these byte[] arrays are allocated directly to the LOH, they are above the LOH threshold (85,000 bytes). At this point there are a few possible solutions:-

Use System.Buffers to rent two byte[] arrays from a pool of byte[] arrays
Use Microsoft.IO.RecycableMemoryStream and operate directly on the incoming stream using a pool of Stream's
Expose DefaultChunkSize so that consumers of the API can set it themselves
Lower DefaultChunkSize to a number that is below LOH threshold (85,000 bytes)

The first and second solutions are probably the ones with the biggest wins to be had, but that would require a large pull request and introducing a dependency that the library maintainers might not want². The third solution means that the consumers of the library have to know about the problem and set it to a reasonable number to avoid LOH allocations. No, it seems like the fourth solution is the most likely to get accepted and has the least possibility of breaking existing functionality.

All we need is a number that is lower than 85,000, normally something like 84,000 would have been perfectly suitable. However, a few weeks prior to discovering this problem I was poking around Reference Source (investigating a different issue) when I stumbled across this gem:-

// We pick a value that is the largest multiple of 4096 that is still smaller than the large object heap threshold (85K).
// The CopyTo/CopyToAsync buffer is short-lived and is likely to be collected at Gen0, and it offers a significant
// improvement in Copy performance.
private const int _DefaultCopyBufferSize = 81920;

Windows memory pages are 4,096 bytes in size, so picking a multiple of that which falls under the LOH threshold (85,000 bytes) makes complete sense. Time to fork, branch, create an issue, and make a pull request.

Luckily, we can make the change locally³ and see what the benefits are. Statistics for one hundred uploads of the same image via PutObject:-

Version	Total Memory Allocations	SOH	LOH
1 (DefaultChunkSize at 131,072)	59 MB	33 MB	25 MB
1 (DefaultChunkSize at 81,920)	51 MB	51 MB	0.4 MB

Idle hands

Whilst waiting for my pull request to be reviewed I decided to poke around the AWS S3 documentation and I stumbled across the concept of pre-signed URLs. That sounds interesting! Creating V2 of the uploader:-

public class UploaderV2 : IUploader
{
    private readonly AWSCredentials _credentials;
    private readonly RegionEndpoint _region;
    private readonly string _bucketName;

    public UploaderV2(AWSCredentials credentials, RegionEndpoint region, string bucketName)
    {
        _credentials = credentials;
        _region = region;
        _bucketName = bucketName;
    }

    public void Upload(Stream objectToUpload, int iteration)
    {
        try
        {
            using (var client = new AmazonS3Client(_credentials, _region))
            {
                var request = new GetPreSignedUrlRequest
                {
                    BucketName = _bucketName,
                    Verb = HttpVerb.PUT,
                    Expires = DateTime.Now.AddMinutes(5),
                    ServerSideEncryptionMethod = ServerSideEncryptionMethod.None,
                    Key = "v2/" + iteration + ".png",
                    Headers = { [HeaderKeys.XAmzAclHeader] = S3CannedACL.PublicRead.Value }
                };

                var preSignedUrl = client.GetPreSignedURL(request);
                var webRequest = WebRequest.CreateHttp(preSignedUrl);
                webRequest.Method = WebRequestMethods.Http.Put;
                webRequest.ContentType = "image/*";
                webRequest.AllowWriteStreamBuffering = false;
                webRequest.ContentLength = objectToUpload.Length;
                webRequest.Headers[HeaderKeys.XAmzAclHeader] = S3CannedACL.PublicRead.Value;

                using (var dataStream = webRequest.GetRequestStream())
                using (objectToUpload)
                {
                    objectToUpload.CopyTo(dataStream, objectToUpload.Length > 81920 ? 81920 : (int)objectToUpload.Length);
                }

                using (webRequest.GetResponse())
                {

                }
            }
        }
        catch (Exception exception)
        {
            Console.WriteLine(exception);
        }
    }
}

We see it has the following statistics when uploading the same file one hundred times:-

Version	Total Memory Allocations	SOH	LOH
1 (DefaultChunkSize at 131,072)	59 MB	33 MB	25 MB
1 (DefaultChunkSize at 81,920)	51 MB	51 MB	0.4 MB
2 (Using pre-signed URLs)	17 MB	17 MB	0.4 MB

That is pretty awesome, and all we actually had to do to achieve that gain was read the documentation! Well, not true, you have the benefit of reading a summarised article with all the juicy bits. The work you see here took place over the course of a week, slotted in between client work.

There is a small downside to using GetPreSignedURL in that if the GetPreSignedUrlRequest is modified and the WebRequest is not modified accordingly then AWS will return HTTP 403 Forbidden (e.g. removing the XAmzAclHeader on the WebRequest). This is because the client-side hash and the server-side hashes no longer match.

Just one more thing

Thanks to my last article I have learnt what nerd sniping is - something I do to myself quite a lot. At this stage I was feeling that giddiness about what else could be shaved off, I was wholly looking at the 0.4 MB remaining on the LOH. Again, dotTrace points us in the direction of code path causing that 0.4 MB allocation to the LOH:-

Yikes, that looks fairly serious. Quietly backing away and trying a different tact; we know a pre-signed URL looks something like this:-

https://##bucket_name##.s3.##region_name##.amazonaws.com/##path##/##file_name##?X-Amz-Expires=300&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=##access_key##/20180613/##region_name##/s3/aws4_request&X-Amz-Date=20180613T233349Z&X-Amz-SignedHeaders=host;x-amz-acl&X-Amz-Signature=6bbcb0f802ad86022674e827d574b7a34a00ba76cd1411016c3581ba27fa5450

We should be able to generate that URL ourselves as AWS has very kindly published their signing process. At this point I will admit that I was ready to accept defeat and just leave the 0.4 MB kicking about on the LOH. I really did not feel like the reams of code I was about to write to possibly eliminate that remaining 0.4 MB was going to be worth it.

That was until I spotted an example of what I wanted. With considerably less effort required on my part; V3 was born:-

Version	Total Memory Allocations	SOH	LOH
1 (DefaultChunkSize at 131,072)	59 MB	33 MB	25 MB
1 (DefaultChunkSize at 81,920)	51 MB	51 MB	0.4 MB
2 (Using pre-signed URLs)	17 MB	17 MB	0.4 MB
3 (Using manually pre-signed URLs)	13 MB	13 MB	0 MB

V3 was just an experiment to see what was possible, given how small the gains and how much code there is to maintain it is not something we would actually use in production code. The discovery of pre-signed URLs is the main win here;-

Version	Total Memory Allocations	SOH	LOH
1 (DefaultChunkSize at 131,072)	59 MB	33 MB	25 MB
2 (Using pre-signed URLs)	17 MB	17 MB	0.4 MB
Percentage	-71.18%	-48.48%	-98.4%

Meanwhile my pull request had been merged and released in version 3.3.21.19 of AWSSDK.Core. Quick overview of the timeline:-

2018-03-07 - Issue created on the aws-sdk-net repository
2018-03-13 - Pull request sent in
2018-03-29 - Pull request merged
2018-03-29 - New version of AWSSDK.Core released to NuGet

I love open source.

TLDR - Give me the good stuff

Versions of AWSSDK.Core below 3.3.21.19 caused a fixed cost of 0.3 MB per invocation of PutObject on the AWS S3 .NET client. This was rectified in versions 3.3.21.19 and above. For particularly hot code paths, it is worth exploring the use of GetPreSignedURL on the AWS S3 .NET client as that dropped LOH allocations by 98% in our context and use case.

Find me Twitter, LinkedIn, or GitHub.

Footnotes

¹ Another reason may be that WinDbg still scares me.

² That being said a recent conversation has been started to take advantage of .NET Core goodness

³ Make sure to build in release unlike a certain someone - okay it was me

Strings Are Evil

Indy Singh — Thu, 14 Jun 2018 21:10:42 +0000

Reducing memory allocations from 7.5GB to 32KB

Context of the problem
Establishing a baseline
Easy win 1
Easy win 2
Splits are never cool
Lists are not always nice
Pooling byte arrays
Goodbye StringBuilder
Skipping commas
The war between classes and structs
Goodbye StreamReader
TLDR - give me a table

Context of the problem

Codeweavers is a financial services software company, part of what we do is to enable our customers to bulk import their data into our platform. For our services we require up-to-date information from all our clients, which includes lenders and manufacturers across the UK. Each of those imports can contain several hundred megabytes uncompressed data, which will often be imported on a daily basis.

This data is then used to power our real-time calculations. Currently this import process has to take place outside of business hours because of the impact it has on memory usage.

In this article we will explore potential optimisations to the import process specifically within the context of reducing memory during the import process. If you want to have a go yourself, you can use this code to generate a sample input file and you can find all of the code talked about here.

Establishing a baseline

The current implementation uses StreamReader and passes each line to the lineParser.

using (StreamReader reader = File.OpenText(@"..\..\example-input.csv"))
{
    try
    {
        while (reader.EndOfStream == false)
        {
            lineParser.ParseLine(reader.ReadLine());
        }
    }
    catch (Exception exception)
    {
        throw new Exception("File could not be parsed", exception);
    }
}

The most naive implementation of a line parser that we originally had looked something like this:-

public sealed class LineParserV01 : ILineParser
{
    public void ParseLine(string line)
    {
        var parts = line.Split(',');

        if (parts[0] == "MNO")
        {
            var valueHolder = new ValueHolder(line);
        }
    }
}

The ValueHolder class is used later on in the import process to insert information into the database:-

public class ValueHolder
{
    public int ElementId { get; }
    public int VehicleId { get; }
    public int Term { get; }
    public int Mileage { get; }
    public decimal Value { get; }

    public ValueHolder(string line)
    {
        var parts = line.Split(',');

        ElementId = int.Parse(parts[1]);
        VehicleId = int.Parse(parts[2]);
        Term = int.Parse(parts[3]);
        Mileage = int.Parse(parts[4]);
        Value = decimal.Parse(parts[5]);
    }
}

Running this example as a command line application and enabling monitoring:-

public static void Main(string[] args)
{
    AppDomain.MonitoringIsEnabled = true;

    // do the parsing

    Console.WriteLine($"Took: {AppDomain.CurrentDomain.MonitoringTotalProcessorTime.TotalMilliseconds:#,###} ms");
    Console.WriteLine($"Allocated: {AppDomain.CurrentDomain.MonitoringTotalAllocatedMemorySize / 1024:#,#} kb");
    Console.WriteLine($"Peak Working Set: {Process.GetCurrentProcess().PeakWorkingSet64 / 1024:#,#} kb");

    for (int index = 0; index <= GC.MaxGeneration; index++)
    {
        Console.WriteLine($"Gen {index} collections: {GC.CollectionCount(index)}");
    }
}

Our main goal today is to reduce allocated memory. In short, the less memory we allocate, the less work the garbage collector has to do. There are three generations that garbage collector operates against, we will also be monitoring those. Garbage collection is a complex topic and outside of the scope of this article; but a good rule of thumb is that short-lived objects should never be promoted past generation 0.

We can see V01 has the following statistics:-

Took: 8,750 ms
Allocated: 7,412,303 kb
Peak Working Set: 16,720 kb
Gen 0 collections: 1809
Gen 1 collections: 0
Gen 2 collections: 0

Almost 7.5 GB of memory allocations to parse a three hundred megabyte file is less than ideal. Now that we have established the baseline, let us find some easy wins…

Easy win 1

Eagle-eyed readers will have spotted that we string.Split(',') twice; once in the line parser and again in the constructor of ValueHolder. This is wasteful, we can overload the constructor of ValueHolder to accept a string[] array and split the line once in the parser. After that simple change the statistics for V02 are now:-

Took: 6,922 ms
Allocated: 4,288,289 kb
Peak Working Set: 16,716 kb
Gen 0 collections: 1046
Gen 1 collections: 0
Gen 2 collections: 0

Great! We are down from 7.5GB to 4.2GB. But that is still a lot of memory allocations for processing a three hundred megabyte file.

Easy win 2

Quick analysis of the input file reveals that there are 10,047,435 lines, we are only interested in lines that are prefixed with MNO of which there are 10,036,466 lines. That means we are unnecessarily processing an additional 10,969 lines. A quick change to V03 to only parse lines prefixed with MNO:-

public sealed class LineParserV03 : ILineParser
{
    public void ParseLine(string line)
    {
        if (line.StartsWith("MNO"))
        {
            var valueHolder = new ValueHolder(line);
        }
    }
}

This means we defer splitting the entire line until we know it is a line we are interested in. Unfortunately this did not save us much memory. Mainly because we are interested in 99.89% of the lines in the file. The statistics for V03:-

Took: 8,375 ms
Allocated: 4,284,873 kb
Peak Working Set: 16,744 kb
Gen 0 collections: 1046
Gen 1 collections: 0
Gen 2 collections: 0

It is time to break out the trusty profiler, in this case dotTrace:-

Strings in the .NET ecosystem are immutable. Meaning that anything we do to a string always returns a brand new copy. Therefore calling string.Split(',') on every line (remember there are 10,036,466 lines we are interested in) returns that line split into several smaller strings. Each line at minimum has five sections we want to process. That means in the lifetime of the import process we create at least 50,182,330 strings..! Next we will explore what we can do to eliminate the use of string.Split(',').

Splits are never cool

A typical line we are interested in looks something like this:-

MNO,3,813496,36,30000,78.19,,

Calling string.Split(',') on the above line will return a string[] containing:-

'MNO'
'3'
'813496'
'36'
'30000'
'78.19'
''
''

Now at this point we can make some guarantees about the file we are importing:-

The length of each line is not fixed
The number of sections that are delimited by a comma are fixed
We only use the first three characters of each line to determine our interest in the line
This means there are five sections we are interested in but the section length is unknown
Sections do not change locations (e.g MNO is always the first section)

Guarantees established, we can now build a short lived index of the positions of all the commas for a given line:-

private List<int> FindCommasInLine(string line)
{
    var list = new List<int>();

    for (var index = 0; index < line.Length; index++)
    {
        if (line[index] == ',')
        {
            list.Add(index);
        }
    }

    return list;
}

Once we know the position of each comma, we can directly access the section we care about and manually parse that section.

private decimal ParseSectionAsDecimal(int start, int end, string line)
{
    var sb = new StringBuilder();

    for (var index = start; index < end; index++)
    {
        sb.Append(line[index]);
    }

    return decimal.Parse(sb.ToString());
}

private int ParseSectionAsInt(int start, int end, string line)
{
    var sb = new StringBuilder();

    for (var index = start; index < end; index++)
    {
        sb.Append(line[index]);
    }

    return int.Parse(sb.ToString());
}

Putting it all together:-

public void ParseLine(string line)
{
    if (line.StartsWith("MNO"))
    {
        var findCommasInLine = FindCommasInLine(line);

        var elementId = ParseSectionAsInt(findCommasInLine[0] + 1, findCommasInLine[1], line); // equal to parts[1] - element id
        var vehicleId = ParseSectionAsInt(findCommasInLine[1] + 1, findCommasInLine[2], line); // equal to parts[2] - vehicle id
        var term = ParseSectionAsInt(findCommasInLine[2] + 1, findCommasInLine[3], line); // equal to parts[3] - term
        var mileage = ParseSectionAsInt(findCommasInLine[3] + 1, findCommasInLine[4], line); // equal to parts[4] - mileage
        var value = ParseSectionAsDecimal(findCommasInLine[4] + 1, findCommasInLine[5], line); // equal to parts[5] - value
        var valueHolder = new ValueHolder(elementId, vehicleId, term, mileage, value);
    }
}

Running V04 reveals this statistics:-

Took: 9,813 ms
Allocated: 6,727,664 kb
Peak Working Set: 16,872 kb
Gen 0 collections: 1642
Gen 1 collections: 0
Gen 2 collections: 0

Whoops, that is worse than expected. It is an easy mistake to make but dotTrace can help us here…

Constructing a StringBuilder for every section in every line is incredibly expensive. Luckily it is a quick fix, we constructor a single StringBuilder on the construction of V05 and clear it before each usage. V05 now has the following statistics:-

Took: 9,125 ms
Allocated: 3,199,195 kb
Peak Working Set: 16,636 kb
Gen 0 collections: 781
Gen 1 collections: 0
Gen 2 collections: 0

Phew we are back on the downwards trends. We started at 7.5GB and now we are down to 3.2GB.

Lists are not always nice

At this point dotTrace becomes an essential part of the optimisation process. Looking at V05 dotTrace output:-

Building the short lived index of commas positions is expensive. As underneath any List<T> is just a standard T[] array. The framework takes care of re-sizing the underlying array when elements are added. This is useful and very handy in typical scenarios. However, we know that there are six sections we need to process (but we are only interested in five of those sections), ergo there are at least seven commas we want indexes for. We can optimise for that:-

private int[] FindCommasInLine(string line)
{
    var nums = new int[7];
    var counter = 0;

    for (var index = 0; index < line.Length; index++)
    {
        if (line[index] == ',')
        {
            nums[counter++] = index;
        }
    }

    return nums;
}

V06 statistics:-

Took: 8,047 ms
Allocated: 2,650,318 kb
Peak Working Set: 16,560 kb
Gen 0 collections: 647
Gen 1 collections: 0
Gen 2 collections: 0

2.6GB is pretty good, but what happens if we force the compiler to use byte for this method instead of the compiler defaulting to use int:-

private byte[] FindCommasInLine(string line)
{
    byte[] nums = new byte[7];
    byte counter = 0;

    for (byte index = 0; index < line.Length; index++)
    {
        if (line[index] == ',')
        {
            nums[counter++] = index;
        }
    }

    return nums;
}

Re-running V06:-

Took: 8,078 ms
Allocated: 2,454,297 kb
Peak Working Set: 16,548 kb
Gen 0 collections: 599
Gen 1 collections: 0
Gen 2 collections: 0

2.6GB was pretty good, 2.4GB is even better. This is because an int has a much larger range than a byte.

Pooling byte arrays

V06 now has a byte[] array that holds the index of each comma for each line. It is a short lived array, but it is created many times. We can eliminate the cost of creating a new byte[] for each line by using a recent addition to the .NET ecosystem; Systems.Buffers. Adam Sitnik has a great breakdown on using it and why you should. The important thing to remember when using ArrayPool<T>.Shared is you must always return the rented buffer after you are done using it otherwise you will introduce a memory leak into your application.

This is what V07 looks like:-

public void ParseLine(string line)
{
    if (line.StartsWith("MNO"))
    {
        var tempBuffer = _arrayPool.Rent(7);

        try
        {
            var findCommasInLine = FindCommasInLine(line, tempBuffer);
            // truncated for brevity
        }
        finally
        {
            _arrayPool.Return(tempBuffer, true);
        }
    }
}

private byte[] FindCommasInLine(string line, byte[] nums)
{
    byte counter = 0;

    for (byte index = 0; index < line.Length; index++)
    {
        if (line[index] == ',')
        {
            nums[counter++] = index;
        }
    }

    return nums;
}

And V07 has the following statistics:-

Took: 8,891 ms
Allocated: 2,258,272 kb
Peak Working Set: 16,752 kb
Gen 0 collections: 551
Gen 1 collections: 0
Gen 2 collections: 0

Down to 2.2GB, having started at 7.5GB. It is pretty good, but we are not done yet.

Goodbye StringBuilder

Profiling V07 reveals the next problem:-

Calling StringBuilder.ToString() inside of the decimal and int parsers is incredibly expensive. It is time to deprecate StringBuilder and write our own¹ int and decimal parsers without relying on strings and calling int.parse() / decimal.parse(). According to the profiler this should shave off around 1GB. After writing our own int and decimal parsers V08 now clocks in at:-

Took: 6,047 ms
Allocated: 1,160,856 kb
Peak Working Set: 16,816 kb
Gen 0 collections: 283
Gen 1 collections: 0
Gen 2 collections: 0

1.1GB is a huge improvement from where we were last (2.2GB) and even better than the baseline (7.5GB).

¹Code can be found here

Skipping commas

Until V08 our strategy has been to find the index of every comma on each line and then use that information to create a sub-string which is then parsed by calling int.parse() / decimal.parse(). V08 deprecates the use of sub-strings but still uses the short lived index of comma positions.

An alternative strategy would be to skip to the section we are interested in by counting the number of preceding commas then parse anything after the required number of commas and return when we hit the next comma.

We have previously guaranteed that:-

Each section is preceded by a comma.
And that the location of each section within a line does not change.

This would also means we can deprecate the rented byte[] array because we are no longer building a short lived index:-

public sealed class LineParserV09 : ILineParser
{
    public void ParseLine(string line)
    {
        if (line.StartsWith("MNO"))
        {
            int elementId = ParseSectionAsInt(line, 1); // equal to parts[1] - element id
            int vehicleId = ParseSectionAsInt(line, 2); // equal to parts[2] - vehicle id
            int term = ParseSectionAsInt(line, 3); // equal to parts[3] - term
            int mileage = ParseSectionAsInt(line, 4); // equal to parts[4] - mileage
            decimal value = ParseSectionAsDecimal(line, 5); // equal to parts[5] - value
            var valueHolder = new ValueHolder(elementId, vehicleId, term, mileage, value);
        }
    }
}

Unfortunately V09 does not save us any memory, it does however reduce the time taken:-

Took: 5,703 ms
Allocated: 1,160,856 kb
Peak Working Set: 16,572 kb
Gen 0 collections: 283
Gen 1 collections: 0
Gen 2 collections: 0

Another benefit of V09 is that it reads much more closer to the original implementation.

The war between classes and structs

This blog post is not going to cover the difference or the pros/cons of classes vs structs. That topic has been covered many times. In this particular context, it is beneficial to use a struct. Changing ValueHolder to a struct in V10 has the following statistics:-

Took: 5,594 ms
Allocated: 768,803 kb
Peak Working Set: 16,512 kb
Gen 0 collections: 187
Gen 1 collections: 0
Gen 2 collections: 0

Finally, we are below the 1GB barrier. Also, word of warning please do not use a struct blindly, always test your code and make sure the use case is correct.

Goodbye StreamReader

As of V10 the line parser itself is virtually allocation free. dotTrace reveals where the remaining allocations occur:-

Well this is awkward, the framework is costing us memory allocations. We can interact with the file at a lower-level than a StreamReader:-

private static void ViaRawStream(ILineParser lineParser)
{
    var sb = new StringBuilder();

    using (var reader = File.OpenRead(@"..\..\example-input.csv"))
    {
        try
        {
            bool endOfFile = false;
            while (reader.CanRead)
            {
                sb.Clear();

                while (endOfFile == false)
                {
                    var readByte = reader.ReadByte();

                    // -1 means end of file
                    if (readByte == -1)
                    {
                        endOfFile = true;
                        break;
                    }

                    var character = (char)readByte;

                    // this means the line is about to end so we skip
                    if (character == '\r')
                    {
                        continue;
                    }

                    // this line has ended
                    if (character == '\n')
                    {
                        break;
                    }

                    sb.Append(character);
                }

                if (endOfFile)
                {
                    break;
                }

                var buffer = new char[sb.Length];

                for (int index = 0; index < sb.Length; index++)
                {
                    buffer[index] = sb[index];
                }

                lineParser.ParseLine(buffer);
            }
        }
        catch (Exception exception)
        {
            throw new Exception("File could not be parsed", exception);
        }
    }
}

V11 statistics:-

Took: 5,594 ms
Allocated: 695,545 kb
Peak Working Set: 16,452 kb
Gen 0 collections: 169
Gen 1 collections: 0
Gen 2 collections: 0

Well, 695MB is still better than 768MB. Okay, that was not the improvement I was expecting (and rather anti-climatic). Until, we remember we have previously seen and solved this problem before. In V07 we used ArrayPool<T>.Shared to prevent lots of small byte[]. We can do the same here:-

private static void ViaRawStream(ILineParser lineParser)
{
    var sb = new StringBuilder();
    var charPool = ArrayPool<char>.Shared;

    using (var reader = File.OpenRead(@"..\..\example-input.csv"))
    {
        try
        {
            bool endOfFile = false;
            while (reader.CanRead)
            {
                // truncated for brevity

                char[] rentedCharBuffer = charPool.Rent(sb.Length);

                try
                {
                    for (int index = 0; index < sb.Length; index++)
                    {
                        rentedCharBuffer[index] = sb[index];
                    }

                    lineParser.ParseLine(rentedCharBuffer);
                }
                finally
                {
                    charPool.Return(rentedCharBuffer, true);
                }
            }
        }
        catch (Exception exception)
        {
            throw new Exception("File could not be parsed", exception);
        }
    }
}

The final version of V11 has the following statistics:-

Took: 6,781 ms
Allocated: 32 kb
Peak Working Set: 12,620 kb
Gen 0 collections: 0
Gen 1 collections: 0
Gen 2 collections: 0

Yes, only 32kb of memory allocations. That is the climax I was looking for.

Find me Twitter, LinkedIn, or GitHub.

TLDR - Give me a table

Version	Took (ms)	Allocated (kb)	Peak Working Set (kb)	Gen 0 Collections
01	8,750	7,412,303	16,720	1,809
02	6,922	4,288,289	16,716	1,046
03	8,375	4,284,873	16,744	1,046
04	9,813	6,727,664	16,872	1,642
05	8,125	3,199,195	16,636	781
06	8,078	2,454,297	16,548	599
07	8,891	2,258,272	16,752	551
08	6,047	1,160,856	16,816	283
09	5,703	1,160,856	16,572	283
10	5,594	768,803	16,512	187
11	6,781	32	12,620	0

Forem: Indy Singh

Optimising Image Processing

Contents

Investigating old code

Streaming is winning

Pool your streams

Goodbye System.Drawing, you will not be missed

In the interest of transparency

TLDR - tell me what to do

AWS S3 .NET Client High Memory Usage

Contents

Problem discovery

Why is it a problem?

Introducing the best magic number — 81,920

Idle hands

Just one more thing

TLDR - Give me the good stuff

Footnotes

Strings Are Evil

Contents

Context of the problem

Establishing a baseline

Easy win 1

Easy win 2

Splits are never cool

Lists are not always nice

Pooling byte arrays

Goodbye StringBuilder

Skipping commas

The war between classes and structs

Goodbye StreamReader

TLDR - Give me a table