Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Allow override env for all task in dag #3623

Merged
merged 41 commits into from
May 31, 2024

Conversation

Michaelvll
Copy link
Collaborator

@Michaelvll Michaelvll commented May 31, 2024

This is required by #3611, as we should be able to propagate the secrets / envs across all the tasks, allowing sky jobs launch gpt2-pipeline.yaml --env BUCKET_NAME=my-bucket-name

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: pytest tests/test_smoke.py
  • Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

Copy link
Collaborator

@romilbhardwaj romilbhardwaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @Michaelvll!

sky/cli.py Outdated Show resolved Hide resolved
@romilbhardwaj
Copy link
Collaborator

Also I noticed a typo:

sky launch -c gpt2 gpt2.yaml --gpu A100
it should use --gpus instead of --gpu. Can we also include that fix in this PR?

@Michaelvll Michaelvll requested a review from romilbhardwaj May 31, 2024 16:56
Copy link
Collaborator

@romilbhardwaj romilbhardwaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Michaelvll! Ran some manual tests to check envvar behavior, lgtm.

llm/gpt-2/README.md Outdated Show resolved Hide resolved
mode: COPY

setup: |
cd ~
pip install tqdm tiktoken requests datasets
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this intended?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is not needed for training stage, but only for the data processing stage : )

Co-authored-by: Romil Bhardwaj <[email protected]>
Copy link
Collaborator

@cblmemo cblmemo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this feature @Michaelvll ! It looks good to me ;) I'm wondering if we need to clarify this semantic changes in the documentation, but we could do it in another PR as well. Maybe left an TODO or file an issue for this if we decided to delay it?

llm/gpt-2/gpt2-pipeline.yaml Outdated Show resolved Hide resolved
llm/gpt-2/gpt2-train.yaml Outdated Show resolved Hide resolved
sky/utils/dag_utils.py Show resolved Hide resolved
sky/cli.py Show resolved Hide resolved
@Michaelvll Michaelvll merged commit e620ccc into master May 31, 2024
20 checks passed
@Michaelvll Michaelvll deleted the override-env-for-all-task-in-dag branch May 31, 2024 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants