Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDK] test: Add e2e test for tune function. #2399

Merged
merged 23 commits into from
Aug 6, 2024

Conversation

Electronic-Waste
Copy link
Member

What this PR does / why we need it:

  • Adds an e2e test in the run_e2e_experiment.py for the tune function.
  • Change the error field name from metrics_collector to metrics_collector_spec

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2383

Checklist:

  • Docs included if any changes are user facing

@Electronic-Waste
Copy link
Member Author

PTAL👀 @andreyvelich @johnugeorge @tenzen-y @tariq-hasan

Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for doing this @Electronic-Waste!
This should help us to make this API more stable.

test/e2e/v1beta1/scripts/gh-actions/run-e2e-experiment.sh Outdated Show resolved Hide resolved
@Electronic-Waste
Copy link
Member Author

Now run-e2e-tune-api.py has been added, and the error in CI/CD test is addressed.

Also, I found function verify_experiment_results is used in both the tune API test and the original test. So I move this function to verify.py and import it separately in run-e2e-tune-api.py and run-e2e-experiment.py.

Could you take a look?👀 @andreyvelich

@Electronic-Waste
Copy link
Member Author

@andreyvelich I've added a new workflow file e2e-test-tune-api.yaml to .github/workflows, and also, e2e-test-tune-api.sh to test/e2e/v1beta1/scripts/gh-actions. Now, the test for tune API is separate from other tests.

Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good, thank you @tenzen-y, just a few comments.
/assign @johnugeorge @tenzen-y

test/e2e/v1beta1/scripts/gh-actions/build-load.sh Outdated Show resolved Hide resolved
test/e2e/v1beta1/scripts/gh-actions/build-load.sh Outdated Show resolved Hide resolved
test/e2e/v1beta1/scripts/gh-actions/build-load.sh Outdated Show resolved Hide resolved
test/e2e/v1beta1/scripts/gh-actions/run-e2e-tune-api.sh Outdated Show resolved Hide resolved
@Electronic-Waste
Copy link
Member Author

This PR is also related to my GSoC project.

ref issue: #2340

@Electronic-Waste
Copy link
Member Author

I've removed the time.sleep and refactored the workflow file with template.

PTAL👀 @tenzen-y if you are available.

.github/workflows/e2e-test-tune-api.yaml Outdated Show resolved Hide resolved
experiments: ""
# Comma Delimited
trial-images: ""
tune-api: true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tenzen-y Does it look to you on how we separate the template-e2e-test ?

test/e2e/v1beta1/scripts/gh-actions/build-load.sh Outdated Show resolved Hide resolved
@Electronic-Waste
Copy link
Member Author

@andreyvelich I've modified this PR as you suggested in the last few comments. Could you plz take a look when you are available? Thanks so much!

Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for doing this @Electronic-Waste!
It should improve tune API stability.
/lgtm
/approve

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit b6f7cfd into kubeflow:master Aug 6, 2024
63 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add e2e test for tune in the katib_client module
4 participants