[Bug] - ListObjectsV2 - invalid continuation token behaves like v1 marker rather than returning error

[lrm25]

Describe the bug

Direct to s3, an invalid continuation token for ListObjectsV2 will return an error like the following:

<Error><Code>InvalidArgument</Code><Message>The continuation token provided is incorrect</Message><ArgumentName>continuation-token</ArgumentName><RequestId>0NR813BK2X6H3A8Q</RequestId><HostId>l1Qt3CCVIyl/8bibjswShJQpy8SRjGwwEAGoZJ+zl4FrWqHGeC6f2QZpFGABMQ0PuCXMAtt6iKAr3yvRyUJJFA9JfI4At8WQCehon58yRbc=</HostId></Error>

However, via versitygw, something similar to the following is returned:

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>versity-gwtest-bucket-one-1</Name><ContinuationToken>aWrongToken2</ContinuationToken><NextContinuationToken>test_file</NextContinuationToken><KeyCount>1</KeyCount><MaxKeys>1</MaxKeys><IsTruncated>true</IsTruncated><Contents><LastModified>2024-12-18T19:34:09.506Z</LastModified><ETag>9ca68472951153f7ea2f696b219c56cd</ETag><Key>test_file</Key><Size>10</Size><StorageClass>STANDARD</StorageClass></Contents><EncodingType></EncodingType></ListBucketResult>

Also, if the bogus continuation token is ahead of all other keys in alphabetical order:

<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>versity-gwtest-bucket-one-1</Name><ContinuationToken>wrongToken2</ContinuationToken><KeyCount>0</KeyCount><MaxKeys>1</MaxKeys><IsTruncated>false</IsTruncated><EncodingType></EncodingType></ListBucketResult>

To Reproduce

1. Start versitygw, e.g.: ./versitygw --access=******** --secret=******** --region=us-east-1 --iam-dir=/Users/lukemccrone/devel/versitygw/iam --cert /Users/lukemccrone/devel/versitygw/cert.pem --key /Users/lukemccrone/devel/versitygw/versitygw.pem posix --versioning-dir /tmp/versioning /tmp/gw
2. Create bucket, e.g.: aws --no-verify-ssl s3api create-bucket --bucket versity-gwtest-bucket-one-1
3. Create two objects locally
4. Add objects to bucket, e.g. (for each): aws --no-verify-ssl s3api put-object --body /Users/lukemccrone/devel/versitygw/versity-gwtest-files/test_file --bucket versity-gwtest-bucket-one-1 --key test_file
5. Use script at https://github.com/versity/versitygw/blob/test/rest_list_objects_v2/tests/rest_scripts/list_objects.sh, with MAX_KEYS=1 and VERSION_TWO="TRUE", to retrive the NextContinuationToken. The command should look like this in cURL: curl -ks -w %{http_code} https://localhost:7070/versity-gwtest-bucket-one-1?list-type=2&max-keys=1 -H Authorization: AWS4-HMAC-SHA256 Credential=********/20241218/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=c355bc89bd67bfd4b3d4b7801f2dcd908bc74b1ff3794fa067d768add2025887 -H x-amz-content-sha256: UNSIGNED-PAYLOAD -H x-amz-date: 20241218T193411Z -o /Users/lukemccrone/devel/versitygw/versity-gwtest-files/objects.txt
6. Do the same, but set CONTINUATION_TOKEN to something that doesn't match.

Expected behavior
An error is returned.

Server Version
Version : v1.0.9
Build : bd80166
BuildTime: 2024-12-13_04:08:01PM
Darwin Lukes-MacBook-Pro.local 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:25 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6030 arm64

[niksis02]

The implementation of ListObjectsV2's ContinuationToken in the gateway differs from the one in S3. It is more similar to the NextMarker used in ListObjects.

In S3, the ContinuationToken is generated based on object keys, whereas the gateway directly returns the object key as the ContinuationToken.

I'm unsure if this needs to be changed—@benmcclelland might be able to clarify.