AWSSDK.S3 ListObjectsV2 API returns Size incorrectly every other request for some subset of files +/- 2 bytes

[jzabroski]

Describe the bug

There appears to be some prior bugs around this (see: aws/aws-cli#5216), with Amazon stating the issue is caused by third parties using the old ListObjects API. In this case, we are using ListObjectsV2 with GetObjectAsync and us-east-1 region for all requests.

Every other request, the Size is logged as either 84676 or 84674.

This is coming through a third party bucket we are licensed to data on, but I thought to log a formal public issue here for people confused by this as much as I am, and link the third party to this ticket so they can reference it in a AWS support request. This problem has been happening since 2023-03-29 04:30:11.6484437 UTC, but we only caught it today.

I'm also guessing this has caused extra data charges, given this ~84674-84676 byte file has been re-downloaded 57,265 times and counting.

Expected Behavior

S3 Object Size should not be spurious across ListObjectsV2 requests.

Current Behavior

S3 Object Size is not consistent across multiple requests.

Reproduction Steps

Example C# code - you will need to populate AwsClientConfiguration with the appropriate Region, AccessKey and SecretKey, otherwise this is a http://sscce.org/ example

using Amazon.S3;
    public class AwsClientConfiguration
    {
        public string AccessKey { get; set; }
        public string SecretKey { get; set; }
        public string Region { get; set; }
    }
    public interface IS3ClientFactory
    {
        IAmazonS3 Make();
    }
// helper code to make mocking an AmazonS3Client easier.
public class S3ClientFactory : IS3ClientFactory
{
    private readonly AwsClientConfiguraiton _awsClientConfig;
    private readonly ILog _log;

    public S3ClientFactory(AwsClientConfiguration awsClientConfiguration, ILog log)
    {
        _awsClientConfig = awsClientConfiguration ?? throw new ArgumentNullException(nameof(awsClientConfiguration));
        _log = log ?? throw new ArgumentNullException(nameof(log));
    }
    public IAmazonS3 Make()
    {
        var awsClientConfig = _awsClientConfig;
        if (awsClientConfig?.Region == null)
            throw new ArgumentNullException(nameof(AwsClientConfiguration.Region));

        // Unfortunately, the AmazonS3Client takes a concrete RegionEndpoint class instance, and does not accept an IRegionEndpoint interface.
        // Note, some Amazon code samples online still use Amazon.Internal.RegionEndpointProviderV2.GetBySystemName
        // but the API was updated to Amazon.RegionEndpoint.GetBySystemName
        var region = RegionEndpoint.GetBySystemName("us-east-1");
        AmazonS3Client client;
        if (!string.IsNullOrWhiteSpace(awsClientConfig.AccessKey))
        {
            var awsCredentials = new BasicAWSCredentials(awsClientConfig.AccessKey, awsClientConfig.SecretKey);
            client = new AmazonS3Client(awsCredentials, region);
        }
        else
        {
            throw new NotImplementedException("Only AccessKey/SecretKey is supported");
        }

        return client;
    }

// helper code to make mocking an AmazonS3Client easier.
var s3ClientFactory = new S3ClientFactory();

using (var client = s3ClientFactory.Make())
{
    var listRequest = new ListObjectsV2Request
    {
        BucketName = "bucketName",
        MaxKeys = 1000,
        Prefix = "prefix"
    };

    ListObjectsV2Response response;
    do
    {
        try
        {
            response = await client.ListObjectsV2Async(listRequest);
        }
        catch (Exception ex)
        {
            _log.Error(ex.Message, ex);
            throw;
        }

        var message = $"Found {response.S3Objects.Count} items in bucket {listRequest.BucketName} with prefix {listRequest.Prefix}.";

        var s3Files = response.S3Objects;

        var filesToDownload = s3Files.ToList();

        foreach (var fileToDownload in filesToDownload)
        {
            var s3File = fileToDownload;
           _log.Info($"{nameof(s3File.Size)}=[{s3File.Size}]");
// Implementation not provided for ProcessEntry, as it is irrelevant to repro, so commenting out
            //await ProcessEntry(client, s3File, System.IO.Path.GetTempPath());
        }

        listRequest.ContinuationToken = response.NextContinuationToken;
    } while (response.IsTruncated);
}

Possible Solution

No idea - requires S3 Amazon Engineers to investigate.

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.5.3.2

Targeted .NET Platform

.NET 6

Operating System and version

Microsoft Windows Server 2016 Datacenter

[jzabroski]

Upgrading to the latest AWSSDK.S3 seems to fix the issue in local testing. I do not know why, as I don't see any release notes or Pull Request history indicating issues with these two methods. I will deploy the fix to PROD tonight, and if it seems to go away over the next 3 days, will close this request.

[jzabroski]

The problem is still happening on the latest AWSSDK.S3 nuget package. https://www.nuget.org/packages/AWSSDK.S3/3.7.308.5

Strange that I could not reproduce the problem locally.

Locally, i have:

dotnet --list-runtimes

which returns:

Microsoft.NETCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

On different servers, I have:

Microsoft.NETCore.App 6.0.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Microsoft.NETCore.App 6.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Important to note, although I am using async-await interface, my usage is serial, so the thread-safety issues with .NET 6 HttpClient members should not be relevant here if applicable.

[jzabroski]

I've asked our IT department to patch the .NET runtime on these servers to rule out the runtime as the root cause of this bug.

[jzabroski]

We upgraded the .NET Runtime to 6.0.31 and the bug still exists. Please do the needful and revert back if you need any confidential information with guidance on how to confidentially provide the exact bucket and prefix values.

[jzabroski]

Closing. Sorry - this is a bug on our end. This file had more than one version, and the previous version had more bytes than the previous version. When writing the new version, we were using the old file handle, which left 2 trailing bytes from the end of the previous file. Moving to use GetObjectResponse.WriteResponseStreamToFileAsync(string filePath, bool append, CancellationToken cancellationToken) helper method solves the issue.

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.