Stream must be replayable to calculate a body hash

[lon9man]

Describe the bug

Hello,
trying to upload file to S3 in Jetpack Compose.
Uri for file i got from PhotoPicker.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected behavior

file uploaded successfully

Current behavior

error: java.lang.IllegalArgumentException: Stream must be replayable to calculate a body hash

Steps to Reproduce

1. click button "Open Gallery"

@Composable
fun GalleryScreen() {
    var selectedUris by remember {
        mutableStateOf<List<Uri>>(emptyList())
    }

    val multiplePhotoPickerLauncher =
        rememberLauncherForActivityResult(contract = ActivityResultContracts.PickMultipleVisualMedia(),
            onResult = { uris ->
                selectedUris = uris
            })

    fun onOpenGallery() {
        multiplePhotoPickerLauncher.launch(
            PickVisualMediaRequest(ActivityResultContracts.PickVisualMedia.ImageAndVideo)
        )
    }

    Row() {
          Button(onClick = ::onOpenGallery) {
              Text(text = "Open Gallery", color = MaterialTheme.colorScheme.onError)
          }
    }
}

2. prepare additional parameters (bucket, key, metadata) and run fun upload

private suspend fun upload(bucket: String, key: String, metadata: Map<String, String>, uri: Uri) {
    val inputStream = context.contentResolver.openInputStream(uri)
        ?: throw IllegalArgumentException("Unable to open input stream for URI: ${config.uri}")

    try {
        inputStream.use { stream ->
            S3Client.use { s3 ->
                s3.putObject {
                    bucket = bucket
                    key = key
                    metadata = metadata
                    body = ByteStream.fromInputStream(stream)
                }
            }
        }
    } catch (e: Exception) {
        Log.d("TAG", "$e")
        throw e
    }
}

Possible Solution

No response

Context

No response

AWS SDK for Kotlin version

2.0.21

Platform (JVM/JS/Native)

Native

Operating system and version

Ubuntu 20.04.5 LTS

[lauzadis]

ByteStream.fromInputStream needs to assume that the given stream is not replayable because any given InputStream may or may not be replayable.

If you have direct access to the file through the uri, you can set the body to: File(uri).asByteStream(), which does not use an InputStream and is correctly marked as replayable.

We can only support replayable bodies for standard SigV4 signing because we need to include the checksum of the body in the request. Calculating the checksum of a non-replayable body would fully consume it, leaving us unable to send it in the request itself.

[lauzadis]

I've proposed a small fix to infer the replayability of a given InputStream using markSupported(), it won't help you in this case because I believe context.contentResolver.openInputStream returns a FileInputStream which is still not replayable.

smithy-lang/smithy-kotlin#1187

[lauzadis]

We've merged the fix I noted above and it should be available in today's release. Do you have any other questions / concerns?

edit: Fix will be available in Monday's release

[lon9man]

@lauzadis

QUESTIONS

I've proposed a small fix to infer the replayability of a given InputStream using markSupported(), it won't help you in this case because I believe context.contentResolver.openInputStream returns a FileInputStream which is still not replayable.

smithy-lang/smithy-kotlin#1187

1. how this fix should be used? or it is some internal logic, which will be used automatically and will show some error for developer?

If you have direct access to the file through the uri, you can set the body to: File(uri).asByteStream(), which does not use an InputStream and is correctly marked as replayable.

3. AFAIU uri doesn't have direct access to file. i tried next code to copy into real file. after that it works. BUT maybe exists some better approach to make it without additional copy?

fun fileFromContentUri(context: Context, contentUri: Uri): File {
    val fileExtension = getFileExtension(context, contentUri)
    val fileName = "temporary_file" + if (fileExtension != null) ".$fileExtension" else ""

    val tempFile = File(context.cacheDir, fileName)
    tempFile.createNewFile()

    try {
        val oStream = FileOutputStream(tempFile)
        val inputStream = context.contentResolver.openInputStream(contentUri)

        inputStream?.let {
            copy(inputStream, oStream)
        }

        oStream.flush()
    } catch (e: Exception) {
        e.printStackTrace()
    }

    return tempFile
}

thanks!

[lauzadis]

1. The fix affects internal logic, you don't need to modify your application code to get the change.
2. You're right, it's very difficult to get a File handle from content URI in Android, I couldn't find anything to suggest. One option is to continue using openInputStream(uri) and then wrap that in a BufferedInputStream which is now supported after that fix I linked earlier.

[lon9man]

@lauzadis

One option is to continue using openInputStream(uri) and then wrap that in a BufferedInputStream which is now supported after that fix I linked earlier.

hmm.. i installed aws-sdk-kotlin 1.3.82. using your quote above changed code to:

private suspend fun upload(config: UploadConfiguration) {
        val inputStream = context.contentResolver.openInputStream(config.uri)
            ?: throw IllegalArgumentException("Unable to open input stream for URI: ${config.uri}")

        //var tempFile: File? = null

        try {
            //tempFile = fileFromContentUri(context, config.uri)

            BufferedInputStream(inputStream).use { stream ->
                s3Client.get().putObject {
                    bucket = config.bucket
                    key = config.key
                    metadata = config.meta.getUserMetadata().mapKeys { it.key.code }
                    body = ByteStream.fromInputStream(stream)
                    //body = ByteStream.fromFile(tempFile)
                }
            }
        } catch (e: Exception) {
            throw e
        } finally {
            //if (tempFile != null && tempFile.exists()) tempFile.delete()
        }
}

got new error message (without replayable words) in catch:
A header you provided implies functionality that is not implemented, Request ID: V8MVPD3NVC71C71X, Extended request ID: kUD3E3jwhbqNKuKg20dkzOqfPU5I9TubxopNiCi7w6gbMQZVXY7H1/NCYe8z04da3sP62QOsQwBdgwTljbu89UfwWEaQLrAT

am i doing something wrong?

[lauzadis]

Are you using a 3rd party object storage like MinIO? That's the only time we've seen errors similar to this. Can you send a full request wire log by configuring LogMode.LogRequestWithBody on your client?

[lauzadis]

I was able to replicate your error, I don't think BufferedInputStream was the right recommendation. I'm taking a look to see if there's anything better for you to use

[lon9man]

@lauzadis

I'm taking a look to see if there's anything better for you to use

thanks for the help.

Are you using a 3rd party object storage like MinIO
no

next is FYI, because behavior is strange)

Can you send a full request wire log by configuring LogMode.LogRequestWithBody on your client

ohh) it was magic trip. i spent several hours to enable logging.
this docs doesn't work for API < 26.
for API 26 it also needs to make adjustments for packaging.resources.excludes (i didn't finish, maybe something else also).
maybe it should be some warnings in docs about possible challenges?

then i installed logback. i ran sources like in previous post.

so my results of wire log. also interesting)

1. log is enabled

I  18:13:06.791 [DefaultDispatcher-worker-3] DEBUG aws.smithy.kotlin.runtime.http.operation.AuthHandler -rpc="S3.PutObject" sdkInvocationId="60bde336-37f1-421a-bfa1-1869fd252a41"- resolved endpoint: Endpoint(uri=https://s3.us-east-1.amazonaws.com/upload.appname.com, headers=null, attributes={AttributeKey(aws.smithy.kotlin#endpointAuthSchemes)=[AuthOptionImpl(schemeId=AuthSchemeId(id=aws.auth#sigv4), attributes={AttributeKey(aws.smithy.kotlin.signing#AwsSigningRegion)=us-east-1, AttributeKey(aws.smithy.kotlin.signing#AwsSigningService)=s3, AttributeKey(aws.smithy.kotlin.signing#UseDoubleUriEncode)=false})]})
I  18:13:06.794 [DefaultDispatcher-worker-3] DEBUG aws.smithy.kotlin.runtime.http.interceptors.FlexibleChecksumsRequestInterceptor -rpc="S3.PutObject" sdkInvocationId="60bde336-37f1-421a-bfa1-1869fd252a41"- no checksum algorithm specified, skipping flexible checksums processing
I  18:13:06.810 [DefaultDispatcher-worker-3] DEBUG aws.smithy.kotlin.runtime.auth.awssigning.DefaultAwsSignerImpl -rpc="S3.PutObject" sdkInvocationId="60bde336-37f1-421a-bfa1-1869fd252a41"- Calculated signature: 6a35daec350518be878dd88e2e784c9374f1767d5ff6617c83b43e6f3c091aab
I  18:13:06.820 [DefaultDispatcher-worker-3] DEBUG aws.smithy.kotlin.runtime.awsprotocol.ClockSkewInterceptor -rpc="S3.PutObject" sdkInvocationId="60bde336-37f1-421a-bfa1-1869fd252a41"- service did not return "Date" header, skipping skew calculation
I  18:13:06.824 [DefaultDispatcher-worker-3] DEBUG aws.smithy.kotlin.runtime.http.middleware.RetryMiddleware -rpc="S3.PutObject" sdkInvocationId="60bde336-37f1-421a-bfa1-1869fd252a41"- request failed with non-retryable error
D  UploadWorker: java.io.IOException: Stream closed

2. log is disabled

UploadWorker: aws.sdk.kotlin.services.s3.model.S3Exception: A header you provided implies functionality that is not implemented, Request ID: XNAPR3KWRS6160A5, Extended request ID: dFUeKVH31om2BMX0NAakcZu71j5bXaAawp67I2tXvXiTYBnGMZNkYD9JxG1S0cTCU7QgHpHPGjM=

but we can't see SDK logs, because they are disabled))