Failed downloads for large files due to insufficient ephemeral storage

[manning-ncsa]

I am running nginx-s3-gateway as a Kubernetes Deployment running on VMs with about 30GB free local flash storage each. When people attempt to download large files (for example there is a 290GB data file needed by someone), the pods are evicted with messages like:

The node was low on resource: ephemeral-storage. Threshold quantity: 3112078177, 
available: 1272912Ki. Container nginx was using 29907532Ki, request is 0, 
has larger consumption of ephemeral-storage.

Basically, there is some buffer of data that is getting stored by nginx that becomes too large for the Kubernetes node storage capacity. I have tried setting the PROXY_CACHE_MAX_SIZE = "0g" as suggested in another discussion to disable caching, but this has not altered the behavior.

Is there a way to limit whatever local buffer is being filled? Hopefully this is not some fundamental limitation, where the nginx server must have enough free space to temporarily store each file being proxied.

[dekobon]

Hi there, I do not know the details of your Kubernetes cluster configuration, but I suspect that the file path in which NGINX is writing its cache files is mounted as ephemeral storage. The default path in the S3 Gateway is: /var/cache/nginx/s3_proxy. Can you hop on the instance and do a df -h and see if that directory is mounted as ephemeral storage?

Once you know, you can modify/overwrite the file and point the cache path somewhere else: common/etc/nginx/templates/cache.conf.template
Alternatively, if you want to disable caching entirely, you could just replace that file (cache.conf.template) with a blank file.

Please keep us posted on how this goes.

[manning-ncsa]

The df command in one of the nginx-s3-gateway replica containers yields:

root@desdr-server-s3-79cd6cb84b-klb46:/# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          58G   28G   31G  48% /
tmpfs            64M     0   64M   0% /dev
/dev/sda1        58G   28G   31G  48% /etc/hosts
shm              64M     0   64M   0% /dev/shm
tmpfs            30G   12K   30G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            15G     0   15G   0% /proc/acpi
tmpfs            15G     0   15G   0% /proc/scsi
tmpfs            15G     0   15G   0% /sys/firmware

root@desdr-server-s3-79cd6cb84b-klb46:/# du -sh /var/cache/nginx/s3_proxy
142M    /var/cache/nginx/s3_proxy

The S3 cache settings file looks like:

root@desdr-server-s3-79cd6cb84b-klb46:/# cat /etc/nginx/templates/cache.conf.template
# Settings for S3 cache

proxy_cache_path /var/cache/nginx/s3_proxy
levels=1:2
keys_zone=s3_cache:10m
max_size=$PROXY_CACHE_MAX_SIZE
inactive=$PROXY_CACHE_INACTIVE
use_temp_path=off;

root@desdr-server-s3-79cd6cb84b-klb46:/# echo $PROXY_CACHE_MAX_SIZE
0

root@desdr-server-s3-79cd6cb84b-klb46:/# echo $PROXY_CACHE_INACTIVE
60m

As you can see, the PROXY_CACHE_MAX_SIZE value was already 0 but the behavior was unchanged, so I tried your suggestion of of emptying the file by mounting a ConfigMap and commenting out all the lines, but now the container fails to start with error:

2024/02/12 20:52:24 [emerg] 1#1: "proxy_cache" zone "s3_cache" is unknown in /etc/nginx/nginx.conf:59
nginx: [emerg] "proxy_cache" zone "s3_cache" is unknown in /etc/nginx/nginx.conf:59

[manning-ncsa]

I attempted to use max_size=1g and min_free=5g options on the proxy_cache_path setting. The behavior was the same. It is unclear if those options only support atomic operations on files; in other words, if the file being proxied is 300GB, but the min_free value is set to 5GB, perhaps it will continue to crash the server because the cache manager does not act until the file has been completed loaded?

[manning-ncsa]

I solved it by disabling buffering by adding the line proxy_buffering off! Here is the relevant commit.

Thanks for your help @dekobon ; without your suggestion I doubt I would have figured this out.