Memory Issues / Configuring For Kubernetes

[pinkfloydx33]

The Problem

We are using Magick.NET as part of an ASP.NET application that runs as a Linux Docker containers in Kubernetes. The application creates thumbnails for any images our users might upload. The maximum upload size is 30Mb. Everything has been working generally fine. However once we get a handful of user uploads (usually in short succession or triggered by QA testing) the pod's memory consumption rises and Kubernetes OOM kills the pod.

We run our containers with a memory limit of 350Mi. Because of this, before the OOM's I originally set ResourceLimits.LimitMemory to 35% out of precaution. Once we started getting the OOM kills I lowered it even further to 20% which didn't seem to help. I even increased the container max memory to 700Mi and despite this it eventually gets OOM killed.

I had read some other threads here that suggested making sure you were running as x64 (since we're using the AnyCPU build) and that ResourceLimits were configured properly. So I added some basic console logging to check what ResouceLimits.Memory was actually defaulting to, and this was the output:

IMAGEMAGICK MEMORY DEFAULT: 14653595648

Well, 14653595648 bytes equates to more than 13GB which is well over the memory limits set on the pods. It appears that when running in Kubernetes as a Linux container, the defaults are being taken from the MemTotal entry in /proc/meminfo (or it's equivalent) which equate to the node's resources, and not the pod's.

Here's the relevant portion of /proc/meminfo:

MemTotal:       14310152 kB
MemFree:          808760 kB
MemAvailable:    7831240 kB

Instead, the defaults should be taken from the cgroups (v2) file at /sys/fs/cgroup/memory.max, which in this instance contains

367001600

correlating to the limit of 350Mi that this pod currently has.

Questions

Now that I realize what's happening, I can make changes to both the pod configuration and our application so that I'm able to set ResourceLimits based on the pod's actual memory availability. That leaves me with a couple of questions:

1. Is this a bug in the library?
2. What is the recommended value for ResourceLimits.Memory in this constrained environment (350mi), where the largest images we work with are 30mb?
   • does ResourceLimits.Memory tune the same knobs as ResourceLimits.LimitMemory? I can't use the latter since passing a percentage won't work
3. Should I be setting ResourceLimits.MaxMemoryRequest as well?
   • What's the different between Memory and MaxMemoryRequest. The documentation doesn't really explain it to me
4. Are there any other settings worth tweaking manually?

In case it matters:

• we also have a scratch disk attached to the pods that we assign to the library via MagickNET.SetTempDirectory as we were expecting to spill to disk.
• we are using Magick.NET-Q8-AnyCPU version 14.0.0