You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Setup Details
We are using Amazon Kinesis Client Library (KCL) v3 with the Python MultiLang Daemon and have the following cross-account, multi-region architecture:
Account A (Customer Account) – us-west-2
Hosts an Amazon Connect instance.
Streams Agent Events to Amazon Kinesis Data Streams (KDS) in us-west-2.
Account B (Our AWS Account) – us-east-1
Hosts an ECS Cluster that acts as the KCL Consumer.
We intend to create all DynamoDB tables (checkpointing, worker metrics, and coordinator state) in us-east-1, alongside our ECS cluster.
Issue Observed
Despite our expectation that DynamoDB tables would be created in Account B's region (us-east-1), we have noticed that they are instead being created in us-west-2, the same region as KDS.
This behavior occurs because the regionName attribute is used for both:
Kinesis Data Streams (KDS) region
DynamoDB and CloudWatch metrics region
Currently, there is no way to specify separate regions for KDS and other supporting components (DynamoDB, CloudWatch).
Proposed Enhancement
To provide flexibility for multi-region deployments, we propose introducing a new attribute to differentiate KDS's region from other consumer-side services:
Keep the existing regionName attribute, which remains dedicated to DynamoDB/CloudWatch (aka consumer-side services).
Introduce a new kdsRegionName attribute, specifically for KDS.
Update credential providers to use this differentiation when initializing connections.
Rationale & Considerations
We acknowledge that it is generally not recommended to deploy KDS and its consumers in different regions due to potential latency concerns.
However, in some scenarios (such as ours), this setup is necessary, and having the ability to explicitly configure separate regions would provide greater flexibility.
Question & Request for Guidance
Could we introduce the ability to configure separate region attributes? Specifically:
Keep regionName for DynamoDB/CloudWatch (consumer-side services).
Introduce kdsRegionName for Kinesis Data Streams region.
Would appreciate any insights, guidance, or best practices on how to achieve this configuration effectively.
Thank you for your time and consideration!
The text was updated successfully, but these errors were encountered:
Setup Details
We are using Amazon Kinesis Client Library (KCL) v3 with the Python MultiLang Daemon and have the following cross-account, multi-region architecture:
Account A (Customer Account) – us-west-2
Hosts an Amazon Connect instance.
Streams Agent Events to Amazon Kinesis Data Streams (KDS) in us-west-2.
Account B (Our AWS Account) – us-east-1
Hosts an ECS Cluster that acts as the KCL Consumer.
We intend to create all DynamoDB tables (checkpointing, worker metrics, and coordinator state) in us-east-1, alongside our ECS cluster.
Issue Observed
Despite our expectation that DynamoDB tables would be created in Account B's region (us-east-1), we have noticed that they are instead being created in us-west-2, the same region as KDS.
This behavior occurs because the regionName attribute is used for both:
Kinesis Data Streams (KDS) region
DynamoDB and CloudWatch metrics region
Currently, there is no way to specify separate regions for KDS and other supporting components (DynamoDB, CloudWatch).
Proposed Enhancement
To provide flexibility for multi-region deployments, we propose introducing a new attribute to differentiate KDS's region from other consumer-side services:
Keep the existing regionName attribute, which remains dedicated to DynamoDB/CloudWatch (aka consumer-side services).
Introduce a new kdsRegionName attribute, specifically for KDS.
Update credential providers to use this differentiation when initializing connections.
Rationale & Considerations
We acknowledge that it is generally not recommended to deploy KDS and its consumers in different regions due to potential latency concerns.
However, in some scenarios (such as ours), this setup is necessary, and having the ability to explicitly configure separate regions would provide greater flexibility.
Question & Request for Guidance
Could we introduce the ability to configure separate region attributes? Specifically:
Keep regionName for DynamoDB/CloudWatch (consumer-side services).
Introduce kdsRegionName for Kinesis Data Streams region.
Would appreciate any insights, guidance, or best practices on how to achieve this configuration effectively.
Thank you for your time and consideration!
The text was updated successfully, but these errors were encountered: