Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Separate Region Configuration for KDS and Other KCL Components (DynamoDB, CloudWatch) in KCL v3 Python #1438

Open
sparishi opened this issue Feb 9, 2025 · 0 comments

Comments

@sparishi
Copy link

sparishi commented Feb 9, 2025

Setup Details
We are using Amazon Kinesis Client Library (KCL) v3 with the Python MultiLang Daemon and have the following cross-account, multi-region architecture:

Account A (Customer Account) – us-west-2

Hosts an Amazon Connect instance.
Streams Agent Events to Amazon Kinesis Data Streams (KDS) in us-west-2.
Account B (Our AWS Account) – us-east-1

Hosts an ECS Cluster that acts as the KCL Consumer.
We intend to create all DynamoDB tables (checkpointing, worker metrics, and coordinator state) in us-east-1, alongside our ECS cluster.
Issue Observed
Despite our expectation that DynamoDB tables would be created in Account B's region (us-east-1), we have noticed that they are instead being created in us-west-2, the same region as KDS.

This behavior occurs because the regionName attribute is used for both:

Kinesis Data Streams (KDS) region
DynamoDB and CloudWatch metrics region
Currently, there is no way to specify separate regions for KDS and other supporting components (DynamoDB, CloudWatch).

Proposed Enhancement
To provide flexibility for multi-region deployments, we propose introducing a new attribute to differentiate KDS's region from other consumer-side services:

Keep the existing regionName attribute, which remains dedicated to DynamoDB/CloudWatch (aka consumer-side services).
Introduce a new kdsRegionName attribute, specifically for KDS.
Update credential providers to use this differentiation when initializing connections.
Rationale & Considerations
We acknowledge that it is generally not recommended to deploy KDS and its consumers in different regions due to potential latency concerns.
However, in some scenarios (such as ours), this setup is necessary, and having the ability to explicitly configure separate regions would provide greater flexibility.
Question & Request for Guidance
Could we introduce the ability to configure separate region attributes? Specifically:

Keep regionName for DynamoDB/CloudWatch (consumer-side services).
Introduce kdsRegionName for Kinesis Data Streams region.
Would appreciate any insights, guidance, or best practices on how to achieve this configuration effectively.

Thank you for your time and consideration!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant