feat(clp-package): Add support for other AWS authentication methods #743

Eden-D-Zhang · 2025-03-04T19:52:08Z

Description

Todo: Mount .aws only if S3 is enabled, pass temporary credentials to webui correctly

Checklist

The PR satisfies the contribution guidelines.
This is a breaking change and that has been indicated in the PR title, OR this isn't a
breaking change.
Necessary docs have been updated, OR no docs need to be updated.

Validation performed

coderabbitai · 2025-03-04T19:52:15Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

haiqi96

In general, I think your apporach makes sense. we need to agree on where to put the .aws directory config in the config file.

Please also have a table where each row as a authentication method supported by boto3, with the following columns, so we can see the big picture

Whether supported by clp package or not
How should user specify such credentials in boto3 (and maybe they don't need to do anything)
How should user specify such authentication info in clp package
How CLP supports the method (i.e. via env, or mount)
Any extra note(for example, webui viewer support)

It's also worthwhile to think about:
On failure, which component will run into the error and what user will see

components/package-template/src/etc/clp-config.yml

components/job-orchestration/job_orchestration/scheduler/job_config.py

components/job-orchestration/job_orchestration/executor/query/fs_search_task.py

components/job-orchestration/job_orchestration/executor/query/extract_stream_task.py

haiqi96 · 2025-03-06T15:39:19Z

components/package-template/src/etc/clp-config.yml

@@ -99,3 +99,6 @@
 #
 ## Location where logs are stored. It will be created if it doesn't exist.
 #logs_directory: "var/log"
+#
+## Location where the `.aws` directory should be read from


quick question: are we going to support aws credentials that specified in Environmental variable in the same way as boto3?

I think you should prepare a table to indicate what we will support and how they will supported.

haiqi96 · 2025-03-06T15:44:05Z

components/clp-py-utils/clp_py_utils/s3_utils.py

 from clp_py_utils.compression import FileMetadata

 # Constants
 AWS_ENDPOINT = "amazonaws.com"


+def get_session_credentials(aws_profile: str = "default") -> Optional[S3Credentials]:


let's move the default to config instead.

haiqi96 · 2025-03-06T15:45:58Z

components/clp-py-utils/clp_py_utils/s3_utils.py

+    return S3Credentials(
+        access_key_id=credentials.access_key,
+        secret_access_key=credentials.secret_key,
+        session_token=credentials.token,


In the case when aws is not configured, i.e. user doesn't configure any profile via aws cli, but instead manually creates a .aws/credentials. do we get back the long term key and a token==null?

A further question is what if user specifies a token in the .aws/credentials under the default profile.

The user could technically manually create any profile with any (at least of what we support) authentication method and provided it's done correctly, it will work. If the profile is just long-term credentials, the credentials.token field would be None. If the use chooses to use a profile containing short-term credentials, boto3 would treat it like any other credentials and try to use it, and fail if it can't authenticate.

In short, it's on the user to configure the credentials properly, with or without the aws cli.

If the use chooses to use a profile containing short-term credentials, boto3 would treat it like any other credentials and try to use it, and fail if it can't authenticate.

In that case, will the returned credentials.token be the same as the one specified in the .aws, or will boto3 generate a new temporary token?

I believe it would be the same as specified

Would you please test the behavior?

Ideally, we should prevent user from using temporary credentials in the profile to start clp package, since eventually it will expire.. so maybe we should return an error in this case.

However, i believe if user just configures their profile through sso, we will always get a temporary credentials (which will be updated whenever we query the credentials from the session)?

components/clp-py-utils/clp_py_utils/s3_utils.py

haiqi96 · 2025-03-06T16:00:24Z

components/job-orchestration/job_orchestration/executor/query/fs_search_task.py

-        }
+        env_vars: Dict[str, str] = None
+
+        if s3_config.credentials is not None:


if it's not in your todo yet, let's factor this into a helper method if possible

yeah, I was thinking it would be a good idea

components/clp-package-utils/clp_package_utils/scripts/start_clp.py

Eden-D-Zhang · 2025-03-06T22:03:36Z

Ignore latest commit doesn't work atm

Eden-D-Zhang added 15 commits March 4, 2025 15:37

Add profile to s3credentials

b4d80e6

Add support for aws profile in s3 utils

1cfd858

typo

10d3cfb

Remove trailing whitespace

a325628

Add ~/.aws directory to clp_config

d95333c

Add aws_config_dir mount to scheduler and worker containers

46c2269

Add function to retrieve credentials from session started from profile

5293897

Add session token to S3Credentials

c1e7ddc

Add temporary credential support to extract_stream

a3230a2

Add temporary credentials to fs_search_task, check for no credentials

88d6f03

Fix typo

4499dc5

Lint fixes

dc272a7

Add support to compress from S3

11d0f7c

Lint fixes

939eee4

Add session token to webui env vars **TEMPORARY**

1e3a3cb

Eden-D-Zhang added 3 commits March 5, 2025 17:31

Mount aws config only if S3 is enabled

e07c0fc

Lint

e97a78c

Add support for long-term profile

add24d2

haiqi96 reviewed Mar 6, 2025

View reviewed changes

components/clp-package-utils/clp_package_utils/scripts/start_clp.py Show resolved Hide resolved

Implement review suggestions

1a6142c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(clp-package): Add support for other AWS authentication methods #743

feat(clp-package): Add support for other AWS authentication methods #743

Eden-D-Zhang commented Mar 4, 2025 •

edited

Loading

coderabbitai bot commented Mar 4, 2025 •

edited

Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

haiqi96 left a comment

haiqi96 Mar 6, 2025

haiqi96 Mar 6, 2025

haiqi96 Mar 6, 2025

Eden-D-Zhang Mar 6, 2025

haiqi96 Mar 6, 2025

Eden-D-Zhang Mar 6, 2025

haiqi96 Mar 6, 2025

haiqi96 Mar 6, 2025

Eden-D-Zhang Mar 6, 2025

Eden-D-Zhang commented Mar 6, 2025

feat(clp-package): Add support for other AWS authentication methods #743

Are you sure you want to change the base?

feat(clp-package): Add support for other AWS authentication methods #743

Conversation

Eden-D-Zhang commented Mar 4, 2025 • edited Loading

Description

Checklist

Validation performed

coderabbitai bot commented Mar 4, 2025 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

haiqi96 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eden-D-Zhang commented Mar 6, 2025

Eden-D-Zhang commented Mar 4, 2025 •

edited

Loading

coderabbitai bot commented Mar 4, 2025 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)