-
Notifications
You must be signed in to change notification settings - Fork 868
Fixed issue calling UploadPart with an unseekable stream and disablin… #4048
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: development
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Fixes an issue where the UploadPart operation fails when using an unseekable stream with DisablePayloadSigning set to true. The fix replaces the use of PartialWrapperStream (which requires seekable streams) with the GetStreamWithLength method that returns a PartialReadOnlyWrapperStream compatible with unseekable streams.
- Updates stream handling in UploadPart to use GetStreamWithLength method for unseekable stream compatibility
- Adds comprehensive integration tests for both PutObject and UploadPart operations with unseekable streams
- Includes proper dev config file with patch version bump
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
File | Description |
---|---|
sdk/test/Services/S3/IntegrationTests/PutUnseekableStreamTests.cs | New integration test file with tests for PutObject and UploadPart using unseekable streams |
sdk/src/Services/S3/Custom/Internal/AmazonS3PostMarshallHandler.cs | Updates UploadPart stream handling to use GetStreamWithLength method instead of PartialWrapperStream |
generator/.DevConfigs/32a12d7c-afc6-4bcf-a2d8-9c49b335b935.json | Dev config file with patch version bump and changelog entry |
StreamWriter writer = File.CreateText("PutObjectFile.txt"); | ||
writer.Write("This is some sample text.!!"); | ||
writer.Close(); | ||
|
Copilot
AI
Oct 14, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The created file 'PutObjectFile.txt' is never used in the test methods and should be removed to avoid creating unnecessary files during test execution.
StreamWriter writer = File.CreateText("PutObjectFile.txt"); | |
writer.Write("This is some sample text.!!"); | |
writer.Close(); |
Copilot uses AI. Check for mistakes.
|
||
var initiateMultipartUploadRequest = new InitiateMultipartUploadRequest | ||
{ | ||
BucketName = bucketName, | ||
Key = "upload-part-unseekable-test.txt" | ||
}; | ||
|
||
var initiateMultipartUploadResponse = await Client.InitiateMultipartUploadAsync(initiateMultipartUploadRequest); | ||
|
||
var uploadPartRequest = new UploadPartRequest | ||
{ | ||
BucketName = bucketName, | ||
Key = "upload-part-unseekable-test.txt", | ||
UploadId = initiateMultipartUploadResponse.UploadId, | ||
PartNumber = 1, | ||
PartSize = stream.Length, | ||
InputStream = stream, | ||
DisablePayloadSigning = true, | ||
IsLastPart = true, | ||
}; | ||
|
||
|
||
var uploadPartResponse = await Client.UploadPartAsync(uploadPartRequest); | ||
|
||
var completeMultipartUploadRequest = new CompleteMultipartUploadRequest | ||
{ | ||
BucketName = bucketName, | ||
Key = "upload-part-unseekable-test.txt", | ||
UploadId = initiateMultipartUploadResponse.UploadId | ||
}; | ||
|
||
completeMultipartUploadRequest.AddPartETags(uploadPartResponse); | ||
|
||
await Client.CompleteMultipartUploadAsync(completeMultipartUploadRequest); | ||
|
||
var getRequest = new GetObjectRequest | ||
{ | ||
BucketName = bucketName, | ||
Key = "upload-part-unseekable-test.txt" | ||
}; | ||
using (var getResponse = await Client.GetObjectAsync(getRequest)) | ||
{ | ||
using (var reader = new StreamReader(getResponse.ResponseStream)) | ||
{ | ||
var content = reader.ReadToEnd(); | ||
Assert.AreEqual("Hello, S3!", content); | ||
} |
Copilot
AI
Oct 14, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add error handling for the multipart upload scenario. If the UploadPart or CompleteMultipartUpload operations fail, the multipart upload should be aborted to avoid leaving incomplete uploads in S3.
var initiateMultipartUploadRequest = new InitiateMultipartUploadRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt" | |
}; | |
var initiateMultipartUploadResponse = await Client.InitiateMultipartUploadAsync(initiateMultipartUploadRequest); | |
var uploadPartRequest = new UploadPartRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt", | |
UploadId = initiateMultipartUploadResponse.UploadId, | |
PartNumber = 1, | |
PartSize = stream.Length, | |
InputStream = stream, | |
DisablePayloadSigning = true, | |
IsLastPart = true, | |
}; | |
var uploadPartResponse = await Client.UploadPartAsync(uploadPartRequest); | |
var completeMultipartUploadRequest = new CompleteMultipartUploadRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt", | |
UploadId = initiateMultipartUploadResponse.UploadId | |
}; | |
completeMultipartUploadRequest.AddPartETags(uploadPartResponse); | |
await Client.CompleteMultipartUploadAsync(completeMultipartUploadRequest); | |
var getRequest = new GetObjectRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt" | |
}; | |
using (var getResponse = await Client.GetObjectAsync(getRequest)) | |
{ | |
using (var reader = new StreamReader(getResponse.ResponseStream)) | |
{ | |
var content = reader.ReadToEnd(); | |
Assert.AreEqual("Hello, S3!", content); | |
} | |
string uploadId = null; | |
try | |
{ | |
var initiateMultipartUploadRequest = new InitiateMultipartUploadRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt" | |
}; | |
var initiateMultipartUploadResponse = await Client.InitiateMultipartUploadAsync(initiateMultipartUploadRequest); | |
uploadId = initiateMultipartUploadResponse.UploadId; | |
var uploadPartRequest = new UploadPartRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt", | |
UploadId = uploadId, | |
PartNumber = 1, | |
PartSize = stream.Length, | |
InputStream = stream, | |
DisablePayloadSigning = true, | |
IsLastPart = true, | |
}; | |
var uploadPartResponse = await Client.UploadPartAsync(uploadPartRequest); | |
var completeMultipartUploadRequest = new CompleteMultipartUploadRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt", | |
UploadId = uploadId | |
}; | |
completeMultipartUploadRequest.AddPartETags(uploadPartResponse); | |
await Client.CompleteMultipartUploadAsync(completeMultipartUploadRequest); | |
var getRequest = new GetObjectRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt" | |
}; | |
using (var getResponse = await Client.GetObjectAsync(getRequest)) | |
{ | |
using (var reader = new StreamReader(getResponse.ResponseStream)) | |
{ | |
var content = reader.ReadToEnd(); | |
Assert.AreEqual("Hello, S3!", content); | |
} | |
} | |
} | |
catch | |
{ | |
if (uploadId != null) | |
{ | |
await Client.AbortMultipartUploadAsync(new AbortMultipartUploadRequest | |
{ | |
BucketName = bucketName, | |
Key = "upload-part-unseekable-test.txt", | |
UploadId = uploadId | |
}); | |
} | |
throw; |
Copilot uses AI. Check for mistakes.
…ng checksum failing.
fb6fe9d
to
5bd7587
Compare
The PR is turning into a bit of a whack a mole experience. We'll keep plugging away but there are currently some tests failing making this change. |
PR in draft mode till full dry run is successul
Description
The PutObject operation does work with an unseekable stream if you set both
DisablePayloadSigning
is set totrue
but the UploadPart which also fail. That is because it is always wrapping the input stream in aPartialWrapperStream
which requires the stream to be seekable. The PR switches the code to use theGetStreamWithLength
method like PutObject which returns back aPartialReadOnlyWrapperStream
instance which does not require the wrapped stream to be seekable.The content-length will be determine by either the stream's length if available and then fallback to
PartSize property of the
UploadPartRequest`.Motivation and Context
#4010
Testing
Added new integ tests
Dry run: pending