Adapt http and s3 tests to new dynamic env

[evgeniiz321]

No description provided.

[evgeniiz321]

@roman-khimov can you please take a look at this error:

[MainThread] 2024-01-18 03:05:54 [INFO] File with size 201327592 bytes has been generated: /home/ezayats/neofs-testcases/TemporaryDir/TestFilesDir/0e09a77c-3892-46ac-a68f-9aff0e1ae045
[MainThread] 2024-01-18 03:05:54 [INFO] Executing command: neofs-cli --config /home/ezayats/neofs-testcases/wallet_config.yml object put --rpc-endpoint 'localhost:42829' --wallet '/home/ezayats/neofs-testcases/TemporaryDir/49049ab9-f66d-41cc-9585-3188a2aa7cd6.json' --cid 'YbrX3ybSoYR42z8xfQ3AuZtYE2qtu3AaSepJZvYb4TA' --file '/home/ezayats/neofs-testcases/TemporaryDir/TestFilesDir/0e09a77c-3892-46ac-a68f-9aff0e1ae045' --attributes 'chapter1=peace,chapter2=war' --no-progress
[MainThread] 2024-01-18 03:06:09 [INFO] Command: neofs-cli --config /home/ezayats/neofs-testcases/wallet_config.yml object put --rpc-endpoint 'localhost:42829' --wallet '/home/ezayats/neofs-testcases/TemporaryDir/49049ab9-f66d-41cc-9585-3188a2aa7cd6.json' --cid 'YbrX3ybSoYR42z8xfQ3AuZtYE2qtu3AaSepJZvYb4TA' --file '/home/ezayats/neofs-testcases/TemporaryDir/TestFilesDir/0e09a77c-3892-46ac-a68f-9aff0e1ae045' --attributes 'chapter1=peace,chapter2=war' --no-progress
Error:
return code: 1
Output: rpc error: finish object stream: rpc error: code = DeadlineExceeded desc = context deadline exceeded

ir logs: ir.txt
sn logs: sn.txt

Does it mean that the vm is too slow to handle large object? Are there any configuration options to have longer timeouts?

[roman-khimov]

File with size 201327592 bytes has been generated

See nspcc-dev/neofs-dev-env#263. It really can take some time to transfer/process.

[evgeniiz321]

@roman-khimov @vvarg229 can't figure out which value I need to use for MaxObjectSize. I tried to use 524288 (as in the old dev env)
But - s3 gw multipart tests fails like this:

2024-01-20T20:54:01.381Z        debug   layer/multipart_upload.go:629   part details    {"reqId": "448ea8a6-3b31-47e8-95df-bb85f0d0212a", "bucket": "01dd13b8-68b0-43e0-98f7-3c8b122657cd", "cid": "9mQ7sGCmvw2XRvXaw7XBQdobpkEeSqFS7cPoF4Y9epZg", "object": "05c5ea0a-d367-46ee-b14a-755ef4c37638", "upload id": "9797313e-3f99-41a3-ac46-3d5ab13409e9", "part numbers": [3, 1, 2], "oids": ["8JWmbx5Cr6AhJ6rTmhLyuxQn5LMdxqKUS63iSRudk859", "5KVwrZx5RmZb7azeUYqVdMgTmyuE8YG2SBrXqtSzKA2E", "27ZYwsHX8eoGyU84pPhx7Vg5GLcVqT13TaicDrcTjPGt"]}
2024-01-20T20:54:01.407Z        error   layer/multipart_upload.go:455   could not put a completed object (multipart upload)     {"uploadID": "9797313e-3f99-41a3-ac46-3d5ab13409e9", "uploadKey": "05c5ea0a-d367-46ee-b14a-755ef4c37638", "error": "read payload chunk: init payload reader for the next part: init full payload range reading via connection pool: header: status: code = 2049 message = object not found"}
2024-01-20T20:54:01.407Z        error   handler/util.go:29      call method     {"status": 500, "request_id": "448ea8a6-3b31-47e8-95df-bb85f0d0212a", "method": "CompleteMultipartUpload", "bucket": "01dd13b8-68b0-43e0-98f7-3c8b122657cd", "object": "05c5ea0a-d367-46ee-b14a-755ef4c37638", "description": "could not complete multipart upload", "error": "InternalError: 500 => We encountered an internal error, please try again.", "uploadID": "9797313e-3f99-41a3-ac46-3d5ab13409e9", "Key": "05c5ea0a-d367-46ee-b14a-755ef4c37638"}

full logs logs.tar.gz

I also tried to increase/decrease this value (524288), but still - multipart tests work only with the default MaxObjectSize =/

[evgeniiz321]

@roman-khimov with s3 gw version v0.29.0 and MaxObjectSize=524288 getting errors during CompleteMultipartUpload:

2024-01-22T22:41:50.838Z	debug	layer/multipart_upload.go:629	part details	{"reqId": "08d39cdb-3da4-4408-907e-861981d43c0b", "bucket": "d1841446-ab13-40a3-b9b0-47fb56f9338b", "cid": "7e9KxkmWUboms4LTvVooRdP72A2vXV2BaAT22bQhx86G", "object": "5dcadab2-e780-426f-9c22-185ab080f06d", "upload id": "a66c3c2a-5f0c-46e0-908a-33871f723066", "part numbers": [2, 4, 1, 5, 3], "oids": ["93k3jWgKHYwHAJkCGNKG7SVRiscs9eiir7PZ3GFL31Fo", "6rYQqK8ZQMWAzrY3ADJ9hcDCLxMGGmsVrLvo6o9yUp6d", "Fje4XfGdmn8asWzNUB11yFUqFT682yq9zFHNwGeuTvdQ", "BchjfSVJjGeTGdZLq7K9j6QyghR7oK1c7G7f7ACsMiXQ", "7ARGzVJJWCjB7ki7gVSPivLTK1kB9bcU9QvCRkqjjmVN"]}
2024-01-22T22:41:50.838Z	info	api/router.go:162	call method	{"status": 200, "host": "localhost:52003", "request_id": "08d39cdb-3da4-4408-907e-861981d43c0b", "method": "ListObjectParts", "bucket": "d1841446-ab13-40a3-b9b0-47fb56f9338b", "object": "5dcadab2-e780-426f-9c22-185ab080f06d", "description": "OK"}
2024-01-22T22:41:51.524Z	debug	layer/multipart_upload.go:629	part details	{"reqId": "3c1c60ef-1bee-44a0-8212-ddfd7318c5aa", "bucket": "d1841446-ab13-40a3-b9b0-47fb56f9338b", "cid": "7e9KxkmWUboms4LTvVooRdP72A2vXV2BaAT22bQhx86G", "object": "5dcadab2-e780-426f-9c22-185ab080f06d", "upload id": "a66c3c2a-5f0c-46e0-908a-33871f723066", "part numbers": [2, 4, 1, 5, 3], "oids": ["93k3jWgKHYwHAJkCGNKG7SVRiscs9eiir7PZ3GFL31Fo", "6rYQqK8ZQMWAzrY3ADJ9hcDCLxMGGmsVrLvo6o9yUp6d", "Fje4XfGdmn8asWzNUB11yFUqFT682yq9zFHNwGeuTvdQ", "BchjfSVJjGeTGdZLq7K9j6QyghR7oK1c7G7f7ACsMiXQ", "7ARGzVJJWCjB7ki7gVSPivLTK1kB9bcU9QvCRkqjjmVN"]}
2024-01-22T22:41:53.668Z	error	layer/multipart_upload.go:455	could not put a completed object (multipart upload)	{"uploadID": "a66c3c2a-5f0c-46e0-908a-33871f723066", "uploadKey": "5dcadab2-e780-426f-9c22-185ab080f06d", "error": "read payload chunk: init payload reader for the next part: init full payload range reading via connection pool: header: status: code = 2049 message = object not found"}
2024-01-22T22:41:53.669Z	error	handler/util.go:29	call method	{"status": 500, "request_id": "3c1c60ef-1bee-44a0-8212-ddfd7318c5aa", "method": "CompleteMultipartUpload", "bucket": "d1841446-ab13-40a3-b9b0-47fb56f9338b", "object": "5dcadab2-e780-426f-9c22-185ab080f06d", "description": "could not complete multipart upload", "error": "InternalError: 500 => We encountered an internal error, please try again.", "uploadID": "a66c3c2a-5f0c-46e0-908a-33871f723066", "Key": "5dcadab2-e780-426f-9c22-185ab080f06d"}

And during GetObject with different MaxObjectSize (for both big (not multipart) and multipart objects), that one occurs randomly on random s3 tests:

2024-01-22T22:57:24.536Z	debug	layer/layer.go:480	get object	{"reqId": "3ca48970-1989-452b-aad1-43e15a5805ae", "bucket": "2c823437-7e34-4755-a014-b405e56ffad6", "cid": "9A4m2esCN1JJpisF79mubymz4voUGDDBDU2mtiHcieCk", "object": "3d3ff2fa-3639-4e52-b0e8-3d1184a35e01", "oid": "9x25bUPcoqTEZ426YNb1Lsby8HuAd9tCZLB4xwqCyF4B"}
2024-01-22T22:57:24.547Z	error	handler/util.go:29	call method	{"status": 500, "request_id": "3ca48970-1989-452b-aad1-43e15a5805ae", "method": "GetObject", "bucket": "2c823437-7e34-4755-a014-b405e56ffad6", "object": "3d3ff2fa-3639-4e52-b0e8-3d1184a35e01", "description": "could not get object", "error": "init object payload reader: init full payload range reading via connection pool: header: status: code = 2049 message = object not found"}

full logs neofs logs-2.zip

Can you please take a look?

[roman-khimov]

Please create an issue for skipped tests.