OOM downloading large files

[max-degterev]

Describe the Bug

I have an upcoming task uploading/downloading large files and it seems Payload will go OOM. Would it be possible to stream data to/from S3 instead of buffering?

Downloads are as easy as passing object.Body.transformToWebStream() straight to new Response(), not sure what the lift is on uploads but I assume skipping buffering for non image files should be feasible?

Alternatively would it be possible to expose storageClient: https://github.com/payloadcms/payload/blob/main/packages/storage-s3/src/index.ts#L108 so that userland code can make a custom call without the need to install the same dependencies and create another instance.

Link to the code that reproduces this issue

https://github.com/payloadcms/payload/blob/main/packages/storage-s3/src/staticHandler.ts#L53

Reproduction Steps

Use s3Adapter. Try to upload/download 1GB file on a system with 512MB RAM

Which area(s) are affected? (Select all that apply)

plugin: cloud-storage

Environment Info

v3.12

[max-degterev]

Hackfix:

import path from 'path';
import * as AWS from '@aws-sdk/client-s3';
import { secureConfig } from '@/config/cloud';
import type { File as S3File } from '@/payload-types';

let secureClient: AWS.S3 | null = null;
const getStorageClient: () => AWS.S3 = () => {
  if (!secureClient) secureClient = new AWS.S3(secureConfig.config);
  return secureClient;
};

export const getSecureFile = async (file: S3File) => getStorageClient().getObject({
  Bucket: secureConfig.bucket,
  Key: path.posix.join(file.prefix!, file.filename!),
});

// somewhere in handlers:
      const s3Response = await getSecureFile(fileData);

      if (!s3Response.Body) return getErrorResponse(ERROR_NOT_FOUND);

      const stream = s3Response.Body.transformToWebStream();

      return new Response(stream, {
        headers: {
          'Content-Disposition': contentDisposition(fileData.filename!),
          'Content-Type': s3Response.ContentType!,
        },
      });

[max-degterev]

Actually come to think of it sharp supports streaming so there are easy performance gains here by avoiding buffering, less memory consumed and less time to process by piping resized upload directly to S3