Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SDK] Fix empty list for env variables and numpy version #2360

Merged
merged 2 commits into from
Jun 18, 2024

Conversation

andreyvelich
Copy link
Member

After this PR: #2304, the tune API doesn't work correct.

@helenxie-bit and @quloos Identified bug when using tune API.
If user doesn't set env_per_trial parameter, the Experiment creation fails with this error:

"message":"admission webhook \"validator.experiment.katib.kubeflow.org\" denied the request:
invalid spec.trialTemplate: unable to convert: /spec/template/spec/containers/0/env - [] to Job

We should prioritise unit test PR for Katib SDK to help us detect invalid SDK: #2325 cc @tariq-hasan

/assign @johnugeorge @tenzen-y @helenxie-bit @quloos

Signed-off-by: Andrey Velichkevich <[email protected]>
Copy link

@andreyvelich: GitHub didn't allow me to assign the following users: helenxie-bit, quloos.

Note that only kubeflow members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

After this PR: #2304, the tune API doesn't work correct.

@helenxie-bit and @quloos Identified bug when using tune API.
If user doesn't set env_per_trial parameter, the Experiment creation fails with this error:

"message":"admission webhook \"validator.experiment.katib.kubeflow.org\" denied the request:
invalid spec.trialTemplate: unable to convert: /spec/template/spec/containers/0/env - [] to Job

We should prioritise unit test PR for Katib SDK to help us detect invalid SDK: #2325 cc @tariq-hasan

/assign @johnugeorge @tenzen-y @helenxie-bit @quloos

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: Andrey Velichkevich <[email protected]>
@google-oss-prow google-oss-prow bot added size/S and removed size/XS labels Jun 17, 2024
@andreyvelich andreyvelich changed the title [SDK] Fix empty list for env variables [SDK] Fix empty list for env variables and numpy version Jun 17, 2024
@andreyvelich
Copy link
Member Author

@kubeflow/wg-training-leads It looks like numpy 2.0 was released yesterday: numpy/numpy#24300.

Since torchvision just installs the latest numpy version, I am using numpy==1.26.0 version in our Trial images.

Otherwise, I see the following error from PyTorch:

A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.0 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.

Copy link
Member

@tenzen-y tenzen-y left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
/approve

Thank you!

@google-oss-prow google-oss-prow bot added the lgtm label Jun 18, 2024
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit f8b8d8d into kubeflow:master Jun 18, 2024
60 checks passed
@andreyvelich andreyvelich deleted the sdk-fix-list-env branch June 18, 2024 11:29
andreyvelich added a commit to andreyvelich/katib that referenced this pull request Jun 18, 2024
* [SDK] Fix empty list for env variables

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numpy version in tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
andreyvelich added a commit to andreyvelich/katib that referenced this pull request Jun 18, 2024
* [SDK] Fix empty list for env variables

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numpy version in tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
google-oss-prow bot pushed a commit that referenced this pull request Jun 18, 2024
…branch (#2362)

* Fix TestReconcileBatchJob (#2350)

* update

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* update

Signed-off-by: forsaken628 <[email protected]>

* update

Signed-off-by: forsaken628 <[email protected]>

* update

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* cleanup

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* update

Signed-off-by: forsaken628 <[email protected]>

* use gomock

Signed-off-by: forsaken628 <[email protected]>

---------

Signed-off-by: forsaken628 <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Use cache-dependency-path in actions/setup-go for CI workflow (#2355)

Signed-off-by: forsaken628 <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Replace already closed github.com/golang/mock with go.uber.org/mock (#2357)

* replace gomock

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* revert

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

---------

Signed-off-by: forsaken628 <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Replace gRPC code generation tool from Znly/protoc to Buf  (#2344)

* Replace gRPC code generation tool from Znly/protoc to Buf

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* del build.sh

Signed-off-by: forsaken628 <[email protected]>

* cleanup

Signed-off-by: forsaken628 <[email protected]>

* fix test

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* refine

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

* rm outter yaml

Signed-off-by: forsaken628 <[email protected]>

* fix

Signed-off-by: forsaken628 <[email protected]>

---------

Signed-off-by: forsaken628 <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* Upgrade the protobuf version to >=4.21.12,<5 (#2358)

Signed-off-by: Yuki Iwai <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>

* [SDK] Fix empty list for env variables and numpy version (#2360)

* [SDK] Fix empty list for env variables

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numpy version in tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: forsaken628 <[email protected]>
Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Yuki Iwai <[email protected]>
Co-authored-by: coldWater <[email protected]>
Co-authored-by: Yuki Iwai <[email protected]>
shashank-iitbhu pushed a commit to shashank-iitbhu/katib that referenced this pull request Jun 19, 2024
* [SDK] Fix empty list for env variables

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numpy version in tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
@helenxie-bit
Copy link
Contributor

It works now! Thank you so much! @andreyvelich

shashank-iitbhu pushed a commit to shashank-iitbhu/katib that referenced this pull request Jun 30, 2024
* [SDK] Fix empty list for env variables

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numpy version in tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
shashank-iitbhu pushed a commit to shashank-iitbhu/katib that referenced this pull request Jul 25, 2024
* [SDK] Fix empty list for env variables

Signed-off-by: Andrey Velichkevich <[email protected]>

* Fix numpy version in tests

Signed-off-by: Andrey Velichkevich <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants