UploadPart fails with 500 on the first part of the multipart object

[evgeniiz321]

1. Create multipart upload:

COMMAND: 'aws --no-verify-ssl --no-paginate s3api create-multipart-upload --bucket 5676e6e6-98c3-4d37-b31b-10dd54fca1d9 --key 183b408e-13c3-4a7b-ad30-894a1165faea --endpoint-url https://localhost:56919'
OUTPUT:
 urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "Bucket": "5676e6e6-98c3-4d37-b31b-10dd54fca1d9",
    "Key": "183b408e-13c3-4a7b-ad30-894a1165faea",
    "UploadId": "CPHxp7nGwfxEdyHyyEUK8Fvo8joHfHmi7EprTTn56xjZ"
}

RC: 0
Start / End / Elapsed	 20:58:49.367753 / 20:58:50.026036 / 0:00:00.658283

2. Try to upload first part:

COMMAND: 'aws --no-verify-ssl --no-paginate s3api upload-part --bucket 5676e6e6-98c3-4d37-b31b-10dd54fca1d9 --key 183b408e-13c3-4a7b-ad30-894a1165faea --upload-id CPHxp7nGwfxEdyHyyEUK8Fvo8joHfHmi7EprTTn56xjZ --part-number 1 --body /home/runner/work/neofs-testcases/neofs-testcases/neofs-testcases/TemporaryDir/TestFilesDir/183b408e-13c3-4a7b-ad30-894a1165faea_part_1 --endpoint-url https://localhost:56919'
OUTPUT:
 urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 0): We encountered an internal error, please try again.
RC: 254
Start / End / Elapsed	 20:58:50.652536 / 20:58:51.629319 / 0:00:00.976783

In s3 gw logs I see errors like these:

error	handler/util.go:29	call method	{"status": 500, "request_id": "19611a1a-5376-456d-a650-d956cd5a6b9c", "method": "UploadPart", "bucket": "df4547b0-db3b-4671-a111-305e35c4222c", "object": "32936272-3246-4371-ac45-fbed0a33843c", "description": "could not upload a part", "error": "copy payload with buffer: status: code = 2048 message = access to object operation denied", "uploadID": "8uhUVrYJtW8SkRfGDWFgpwh8VcbfEKyqgxk5mXEW8uv2", "Key": "32936272-3246-4371-ac45-fbed0a33843c"}
error	handler/util.go:29	call method	{"status": 500, "request_id": "3b4c1523-8da4-4c68-a55d-ed86f75f5480", "method": "UploadPart", "bucket": "df4547b0-db3b-4671-a111-305e35c4222c", "object": "32936272-3246-4371-ac45-fbed0a33843c", "description": "could not upload a part", "error": "copy payload with buffer: status: code = 2048 message = access to object operation denied", "uploadID": "8uhUVrYJtW8SkRfGDWFgpwh8VcbfEKyqgxk5mXEW8uv2", "Key": "32936272-3246-4371-ac45-fbed0a33843c"}

Complete logs available here - https://rest.fs.neo.org/HXSaMJXk2g8C14ht8HSi7BBaiYZ1HeWh2xnWPGQCg4H6/444-1719005333/index.html#suites/0865de5bb52b9f1109887595bf1f9535/5058435b934945cd/.

It can be a test issue as well, but right now it is not clear what is the reason.

[carpawell]

See nspcc-dev/neofs-node#2913, it now behaves differently and all object parts seem to be PUT successfully.

[carpawell]

Can confirm my local fails were a local problem with outdated aws cli. @evgeniiz321, you can consider closing this issue.

[roman-khimov]

Fixed in the node, nothing to do for gateway (except for things already done).