[helm] S3 Exceptions: Wrong region and malformed authorization header

[joeybenamy]

Helm Chart Version

1.2.0

What step the error happened?

During the Sync

Relevant information

With Airbyte 1.2.0 and Helm chart 1.2.0, syncs and checks on sources and destinations fail with an HTTP 504. Relevant Helm values:

global:
  storage:
    bucket:
      log: airbyte-logs-uat
      state: airbyte-logs-uat
    s3:
      region: us-east-1

With the Helm chart configured to use S3 storage for logs and state, starting any sync or running a check for any connector causes the below exception. Example:

1. In Airbyte UI, go to Destinations
2. Select the Snowflake connector
3. Under Settings go to "Test the destination"
4. Click "Retest saved destination"
5. Test fails with error An unknown error occurred. (HTTP 504)

Relevant log output

connector-sidecar 2024-11-07 18:27:51,123 [main]    ERROR    i.a.c.ConnectorWatcher(handleException):177 - Error performing operation: io.airbyte.workers.workload.exception.DocStoreAccessException
connector-sidecar io.airbyte.workers.workload.exception.DocStoreAccessException: Unable to write output for 424892c4-daac-4491-b35d-c6688ba547ba_54a33d37-88f3-4af7-b16f-91af2df2fb40_0_check
connector-sidecar     at io.airbyte.workers.workload.JobOutputDocStore.writeOutput(JobOutputDocStore.kt:72)
connector-sidecar     at io.airbyte.workers.workload.JobOutputDocStore.write(JobOutputDocStore.kt:38)
connector-sidecar     at io.airbyte.connectorSidecar.ConnectorWatcher.saveConnectorOutput(ConnectorWatcher.kt:165)
connector-sidecar     at io.airbyte.connectorSidecar.ConnectorWatcher.run(ConnectorWatcher.kt:74)
connector-sidecar     at io.airbyte.connectorSidecar.$ConnectorWatcher$Definition.initialize(Unknown Source)
connector-sidecar     at io.airbyte.connectorSidecar.$ConnectorWatcher$Definition.instantiate(Unknown Source)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2330)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2300)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2312)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3123)
connector-sidecar     at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:3025)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.initializeEagerBean(DefaultBeanContext.java:2698)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.initializeContext(DefaultBeanContext.java:1990)
connector-sidecar     at io.micronaut.context.DefaultApplicationContext.initializeContext(DefaultApplicationContext.java:307)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.configureAndStartContext(DefaultBeanContext.java:3314)
connector-sidecar     at io.micronaut.context.DefaultBeanContext.start(DefaultBeanContext.java:345)
connector-sidecar     at io.micronaut.context.DefaultApplicationContext.start(DefaultApplicationContext.java:215)
connector-sidecar     at io.micronaut.context.ApplicationContextBuilder.start(ApplicationContextBuilder.java:286)
connector-sidecar     at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt:16)
connector-sidecar     at io.airbyte.connectorSidecar.ApplicationKt.main(Application.kt)
connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
connector-sidecar     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
connector-sidecar     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:93)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:50)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:38)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:74)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:43)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:79)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:41)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:55)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:39)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.executeRequest(RetryableStage2.java:93)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:56)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage2.execute(RetryableStage2.java:36)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
connector-sidecar     at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
connector-sidecar     at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
connector-sidecar     at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
connector-sidecar     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
connector-sidecar     at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
connector-sidecar     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
connector-sidecar     at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191)
connector-sidecar     at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:447)
connector-sidecar     at io.airbyte.workers.workload.JobOutputDocStore.writeOutput(JobOutputDocStore.kt:68)
connector-sidecar     ... 20 common frames omitted

[marcosmarxm]

@airbytehq/platform-deployments can someone take a look in this issue?

[marcosmarxm]

@joeybenamy can you confirm the bucket is in us-east-1 and not us-west-1?

connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: 
**The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'** 
(Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)

[joeybenamy]

@joeybenamy can you confirm the bucket is in us-east-1 and not us-west-1?

connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: 
**The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'** 
(Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)

The bucket is in us-east-1.

[joeybenamy]

Added some more detail @marcosmarxm

[bgroff]

@joeybenamy It seems likely that this is an issue with your configuration of AWS. Can you see if setting the region to eu-west-1 resolves this issue? Otherwise, I would create a new bucket in the us-east-1 (double check that is your region in the console) and try again.

[joeybenamy]

@joeybenamy It seems likely that this is an issue with your configuration of AWS. Can you see if setting the region to eu-west-1 resolves this issue? Otherwise, I would create a new bucket in the us-east-1 (double check that is your region in the console) and try again.

So I've troubleshooted this a good bit. I've confirmed that the bucket is in the us-east-1 region and the bucket name matches what I've configured in Airbyte. I've also checked the environment on the Airbyte pods in Kubernetes and AWS_DEFAULT_REGION, STORAGE_BUCKET_LOG, STORAGE_BUCKET_STATE, and STORAGE_TYPE are correctly set. Furthermore, AWS_STS_REGIONAL_ENDPOINTS, AWS_ROLE_ARN, and AWS_WEB_IDENTITY_TOKEN_FILE are set correctly for role assumption. The IAM policy for the role has correct actions allowed for the S3 bucket.

[drewipsonhq]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

[joeybenamy]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

[drewipsonhq]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

@joeybenamy Thank you for the quick response. Are you using the helm chart version pre-1.0 or s3 destination image pre-1.0 to resolve this issue?

[joeybenamy]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

@joeybenamy Thank you for the quick response. Are you using the helm chart version pre-1.0 or s3 destination image pre-1.0 to resolve this issue?

@drewipsonhq Good question. We are using the pre-1.0 helm chart version. We aren't using S3 destination. We are experiencing this issue when writing Airbyte logs and state to an S3 bucket

[drewipsonhq]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

@joeybenamy Thank you for the quick response. Are you using the helm chart version pre-1.0 or s3 destination image pre-1.0 to resolve this issue?

@drewipsonhq Good question. We are using the pre-1.0 helm chart version. We aren't using S3 destination. We are experiencing this issue when writing Airbyte logs and state to an S3 bucket

Ah I thought it could be the state and log bucket since I am also seeing this error on non-s3 related connections (e.g. Hubspot), but I have not had enough time to debug thoroughly. Appreciate the help and feedback.

[joeybenamy]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

@joeybenamy Thank you for the quick response. Are you using the helm chart version pre-1.0 or s3 destination image pre-1.0 to resolve this issue?

@drewipsonhq Good question. We are using the pre-1.0 helm chart version. We aren't using S3 destination. We are experiencing this issue when writing Airbyte logs and state to an S3 bucket

Ah I thought it could be the state and log bucket since I am also seeing this error on non-s3 related connections (e.g. Hubspot), but I have not had enough time to debug thoroughly. Appreciate the help and feedback.

@drewipsonhq You're welcome. I think we're going to need a community contribution to get this fixed. I briefly took a look at the code, but need to look some more. I feel like the fix here is pretty simple, a shame it hasn't been fixed yet. This is the last remaining blocker before we can upgrade Airbyte to 1.x.

[joeybenamy]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

@joeybenamy Thank you for the quick response. Are you using the helm chart version pre-1.0 or s3 destination image pre-1.0 to resolve this issue?

@drewipsonhq Good question. We are using the pre-1.0 helm chart version. We aren't using S3 destination. We are experiencing this issue when writing Airbyte logs and state to an S3 bucket

Ah I thought it could be the state and log bucket since I am also seeing this error on non-s3 related connections (e.g. Hubspot), but I have not had enough time to debug thoroughly. Appreciate the help and feedback.

@drewipsonhq Hey again. I did some digging through the Airbyte code and I think this error is because Airbyte is not trying to use the S3 bucket we are expecting. The bucket it is trying to use is a default, which is located in eu-west-1. Since we are configuring the Airbyte AWS S3 client with us-east-1, this is why we get this specific error. I believe the solution is to set these Helm values to the same value:

• `global.storage.bucket.log: my-bucket-in-us-east-1
• `global.storage.bucket.state: my-bucket-in-us-east-1
• `global.storage.bucket.workloadOutput: my-bucket-in-us-east-1

I am in the process of testing this now. I believe Airbyte is trying to use the airbyte-storage bucket which is in the eu-west-1 region and obviously not a bucket we have access to.

[joeybenamy]

@joeybenamy Did you find a resolution for this error? I am running into the same thing after upgrading to helm version 1.4.0. Already configured S3 destinations (residing in us-east-1) for connection jobs that worked prior to the upgrade are now throwing this error:

Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1'

The buckets are indeed in us-east-1. Configuration variables and IAM permissions reflect this also.

@drewipsonhq Have not found a solution yet. We are still using pre-1.0 due to this issue.

@joeybenamy Thank you for the quick response. Are you using the helm chart version pre-1.0 or s3 destination image pre-1.0 to resolve this issue?

@drewipsonhq Good question. We are using the pre-1.0 helm chart version. We aren't using S3 destination. We are experiencing this issue when writing Airbyte logs and state to an S3 bucket

Ah I thought it could be the state and log bucket since I am also seeing this error on non-s3 related connections (e.g. Hubspot), but I have not had enough time to debug thoroughly. Appreciate the help and feedback.

@drewipsonhq Hey again. I did some digging through the Airbyte code and I think this error is because Airbyte is not trying to use the S3 bucket we are expecting. The bucket it is trying to use is a default, which is located in eu-west-1. Since we are configuring the Airbyte AWS S3 client with us-east-1, this is why we get this specific error. I believe the solution is to set these Helm values to the same value:

• `global.storage.bucket.log: my-bucket-in-us-east-1
• `global.storage.bucket.state: my-bucket-in-us-east-1
• `global.storage.bucket.workloadOutput: my-bucket-in-us-east-1

I am in the process of testing this now. I believe Airbyte is trying to use the airbyte-storage bucket which is in the eu-west-1 region and obviously not a bucket we have access to.

@marcosmarxm @drewipsonhq I spent a few hours yesterday going through the Airbyte platform code and I've discovered the following:

1. When the global storage for Airbyte is configured for S3, the Helm chart still uses airbyte-logs or airbyte-storage S3 bucket names unless you explicitly set the four properties under global.storage.bucket. Since those 2 buckets exist in eu-west-1 and our AWS environment is not in eu-west-1 we get this exact error from AWS:

connector-sidecar Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-1' (Service: S3, Status Code: 400, Request ID: MRAX1XEYMWCP7HGK, Extended Request ID: b8msqNSMDnZE28rVdes2W7eX/ScYpaq/HxnE3o315ABm9rByXNY5nYAH1+NPYDagVXnUi5MeTNM=)

2. When the following Helm values are all set explicitly to the intended bucket, the error is resolved:

global.storage.bucket.log
global.storage.bucket.state
global.storage.bucket.workloadOutput
global.storage.bucket.activityPayload

3. My suggestion would be that when global storage is configured for S3, that the Helm chart requires the above 4 properties to be set explicitly.