-
Notifications
You must be signed in to change notification settings - Fork 531
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support buildkite CICD and restructure smoke tests #4396
Support buildkite CICD and restructure smoke tests #4396
Conversation
b3a5520
to
cd64c4c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for this change which will be super helpful, @zpoint! I left comments primarly on generate_pipeline.py.
I did not review the smoke_tests/ directory, as I understand these were mostly juts copied from the old test_smoke.py. Were there any significant changes in the tests?
@cg505 Thanks for your review! Yes, it's mostly copied, with some private names starting with _ changed to public names. I will restore and copy back the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, looks pretty much good to me. Left a couple more comments. Two additional things:
- I think ideally we will want to move toward using the pytest marks as intended, and split things up/fix the tests to work better in parallel. Then we can avoid this hacky ast parsing method. For now, this is fine.
- Can we look into enforcing lint (pylint, mypy, yapf, etc) on tests/ dir?
Neither of these should be in this PR, just things for the future.
@cg505 Thanks, the comment has been resolved. Yes, I agree:
Both tasks will require time, we can address them in a future PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all the changes! Looks good to me now!
Probably ok to merge this to avoid any more smoke test conflicts with master, and we can figure out the remaining buildkite agent/deployment questions after |
The PR is based on #4386. Please review and merge that PR first, then review and merge this one.test_smoke.py
into multiple files within this directory..buildkite
to contain all the buildkite-related files.test_required_before_merge.py
will be triggered only by valid users' comment on a PR./pre-merge-test aws
//pre-merge-test aws,azure
//pre-merge-test
(default aws).buildkite ├── generate_pipeline.py -- script to generate following yaml automatically ├── pipeline_smoke_tests_pre_merge.yaml -- pre merge smoke tests, only the minimal requried, generated from tests/smoke_tests/test_required_before_merge.py └── pipeline_smoke_tests_release.yaml -- release smoke tests, all smoke tests included, generated from tests/smoke_tests/test_*.py
Support the
LOG_TO_STDOUT
environment variable when executing pytest, so that the output is visible directly on the CI/CD terminal pageSupport the
SKYPILOT_SUPPRESS_SENSITIVE_LOG
environment variable when executing pytest to suppress logs fromprovision.py
, ensuring sensitive information is not exposed during pre-merge tests.Made a minor change to
.pre-commit-config.yaml
after discovering thatpre-commit
wasn't working for files under.buildkite
.Test
Don't have permission for this repo, currently tested under my personal repo
Per PR merged CI
Scheduled CI
A build attempt link
Tested (run the relevant ones):
bash format.sh
pytest tests/test_smoke.py
pytest tests/test_smoke.py::test_fill_in_the_name
conda deactivate; bash -i tests/backward_compatibility_tests.sh