[GSOC] `hyperopt` suggestion service logic update #2412

shashank-iitbhu · 2024-08-21T22:01:36Z

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #2374

Checklist:

Docs included if any changes are user facing

Signed-off-by: Shashank Mittal <[email protected]>

google-oss-prow · 2024-08-21T22:01:41Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tenzen-y · 2024-08-22T18:39:38Z

/area gsoc

…arch_space.py

Signed-off-by: Shashank Mittal <[email protected]>

andreyvelich

Thank you for this @shashank-iitbhu!
I left a few comments

andreyvelich · 2024-09-02T21:49:59Z

.github/workflows/e2e-test-pytorch-mnist.yaml

          - "file-metrics-collector,pytorchjob-mnist"
-          - "median-stop-with-json-format,file-metrics-collector-with-json-format"
+          - "median-stop-with-json-format,file-metrics-collector-with-json-format"


Please add new line.

andreyvelich · 2024-09-02T21:52:08Z

examples/v1beta1/hp-tuning/hyperopt-distribution.yaml

+      feasibleSpace:
+        min: "0.5"
+        max: "0.9"
+        distribution: "logUniform"


Please add other distribution in this example to make sure we will validate them:

normal logNormal

andreyvelich · 2024-09-02T21:53:07Z

pkg/apis/manager/v1beta1/api.proto

-    NORMAL = 2;
-    LOG_NORMAL = 3;
-    DISTRIBUTION_UNKNOWN = 4;
+    DISTRIBUTION_UNKNOWN = 0;


Please keep the same name as for parameter_type

Suggested change

DISTRIBUTION_UNKNOWN = 0;

UNKNOWN_DISTRIBUTION = 0;

Suggested change

DISTRIBUTION_UNKNOWN = 0;

DISTRIBUTION _UNSPECIFIED = 0;

I would like to select the UNSPECIFIED suffix here.
Please see: https://google.aip.dev/126

Make sense, @tenzen-y should we rename other gRPC parameters to UNSPECIFIED ?

Changing released gRPC, it indicates losing backward compatibility.
So, I would like to keep using the existing API for released protocolbuffers API, @andreyvelich WDYT?

Since these gRPC APIs are not exposed to the end-users, do you still think that we should not change the existing APIs ?
It only affects users who build their own Suggestion service.

Since these gRPC APIs are not exposed to the end-users, do you still think that we should not change the existing APIs ?

Almost correct. Additionally, when users keep using the removed Suggestion Services like the Chocolate Suggestion, users face the same problem.

So, can we collect feedback on the dedicated issue outside of here?

Sure, let's followup on this in the issue, and rename it after few months if we don't get any feedback.
@shashank-iitbhu Please can you create an issue to track it ?

Sure, let's followup on this in the issue, and rename it after few months if we don't get any feedback. @shashank-iitbhu Please can you create an issue to track it ?

Sure, I will create a separate issue to track the renaming of other gRPC parameters to UNSPECIFIED.

andreyvelich · 2024-09-02T21:55:09Z

pkg/controller.v1beta1/suggestion/suggestionclient/suggestionclient.go

@@ -533,14 +533,6 @@ func convertParameterType(typ experimentsv1beta1.ParameterType) suggestionapi.Pa

 func convertFeasibleSpace(fs experimentsv1beta1.FeasibleSpace) *suggestionapi.FeasibleSpace {
 	distribution := convertDistribution(fs.Distribution)


Since convertDistribution doesn't return error, I think you can simple add this line in return statement:

return &suggestionapi.FeasibleSpace{ Max: fs.Max, Min: fs.Min, List: fs.List, Step: fs.Step, Distribution: convertDistribution(fs.Distribution), }

andreyvelich · 2024-09-02T21:55:53Z

test/unit/v1beta1/suggestion/test_hyperopt_service.py

+                            name="param-5",
+                            parameter_type=api_pb2.DOUBLE,
+                            feasible_space=api_pb2.FeasibleSpace(
+                                max="5", min="1", list=[], step="0.5", distribution=api_pb2.LOG_UNIFORM)


Please add more tests cases for other hyperopt distributions.

andreyvelich · 2024-09-02T21:57:32Z

pkg/suggestion/v1beta1/hyperopt/base_service.py

-                )
-            elif param.type == DOUBLE:
-                hyperopt_search_space[param.name] = hyperopt.hp.uniform(
+                hyperopt_search_space[param.name] = hyperopt.hp.uniformint(


If parameter is int, why we can't support other distributions like lognormal ?

Distributions like uniform quniform loguniform normal etc return float values. They are designed to sample from a range of values that can take any real number (float), which might not make sense if we're looking for an integer value.
Although we can definitely add support for these distributions when parameter is int also. Should we do this?

@tenzen-y @kubeflow/wg-training-leads @shashank-iitbhu Should we round this float value to int if user wants to use this distribution and int parameter type ?

@tenzen-y @kubeflow/wg-training-leads @shashank-iitbhu Should we round this float value to int if user wants to use this distribution and int parameter type ?

SGTM
Users can specify the double parameter type if they want to compute more exactly.
But, documentation of this restriction for int parameter type would be better.

andreyvelich · 2024-09-02T21:59:24Z

pkg/suggestion/v1beta1/hyperopt/base_service.py

+                else:
+                    hyperopt_search_space[param.name] = hyperopt.hp.uniform(
+                        param.name, float(param.min), float(param.max)
+                    )


I think, we can simplify this if statement by checking if distribution==UNIFORM or UNKNOWN and step is null, we use hyperopt.hp.uniform().

Signed-off-by: Shashank Mittal <[email protected]> validation fix add e2e tests for hyperopt added e2e test to workflow

Signed-off-by: Shashank Mittal <[email protected]>

Signed-off-by: Shashank Mittal <[email protected]> sigma calculation fixed fix parse new arguments to mnist.py

shashank-iitbhu · 2024-09-11T22:11:31Z

pkg/suggestion/v1beta1/hyperopt/base_service.py

+                        )
+                elif param.distribution == api_pb2.NORMAL:
+                    mu = (float(param.min) + float(param.max)) / 2
+                    sigma = (float(param.max) - float(param.min)) / 6


I followed this article to determine the value of sigma from min and max.
cc @tenzen-y @andreyvelich

Signed-off-by: Shashank Mittal <[email protected]>

tenzen-y · 2024-09-19T20:05:48Z

pkg/suggestion/v1beta1/hyperopt/base_service.py

+                        )
+                elif param.distribution == api_pb2.NORMAL:
+                    mu = (float(param.min) + float(param.max)) / 2
+                    sigma = (float(param.max) - float(param.min)) / 6


Suggested change

sigma = (float(param.max) - float(param.min)) / 6

// We consider the normal distribution based on the range of ±3 sigma.

sigma = (float(param.max) - float(param.min)) / 6

shashank-iitbhu · 2024-09-19T20:24:59Z

@tenzen-y I have added two new parameters, weight_decay and dropout_rate, to the Hyperopt example and passed them to mnist.py, but I haven't used them in the Net class yet in the train and test functions. If you check the logs for this e2e test, the maximum value of the loss metrics is an enormously large number. I can't figure out what I'm missing. Also tested this locally.

hyperopt suggestion logic update

f615e3f

Signed-off-by: Shashank Mittal <[email protected]>

google-oss-prow bot requested review from anencore94, gaocegege and johnugeorge August 21, 2024 22:01

google-oss-prow bot added the size/M label Aug 21, 2024

shashank-iitbhu mentioned this pull request Aug 21, 2024

[GSOC] Project 8: Support various Parameter Distribution in Katib #2374

Open

12 tasks

google-oss-prow bot added the area/gsoc label Aug 22, 2024

shashank-iitbhu added 3 commits August 25, 2024 19:35

Merge upstream master and resolve conflicts in base_service.py and se…

a8bc887

…arch_space.py

fix

a67f373

Signed-off-by: Shashank Mittal <[email protected]>

DISTRIBUTION_UNKNOWN enum set to 0 in gRPC api

365c2f5

Signed-off-by: Shashank Mittal <[email protected]>

google-oss-prow bot added size/L and removed size/M labels Aug 26, 2024

andreyvelich reviewed Sep 2, 2024

View reviewed changes

shashank-iitbhu force-pushed the feat/hyperopt-suggestion-service-update branch from 1a7a831 to fddb763 Compare September 10, 2024 16:23

shashank-iitbhu added 7 commits September 10, 2024 22:00

convert parameter method fix

caa2422

Signed-off-by: Shashank Mittal <[email protected]> validation fix add e2e tests for hyperopt added e2e test to workflow

convert feasibleSpace func updated

0f38a51

Signed-off-by: Shashank Mittal <[email protected]>

renamed DISTRIBUTION_UNKNOWN to DISTRIBUTION_UNSPECIFIED

ae9fa34

Signed-off-by: Shashank Mittal <[email protected]>

fix

910a46c

Signed-off-by: Shashank Mittal <[email protected]>

added more test cases for hyperopt distributions

08b01ac

Signed-off-by: Shashank Mittal <[email protected]>

added support for NORMAL and LOG_NORMAL in hyperopt suggestion service

16dc030

Signed-off-by: Shashank Mittal <[email protected]>

added e2e tests for NORMAL and LOG_NORMAL

282f81d

Signed-off-by: Shashank Mittal <[email protected]> sigma calculation fixed fix parse new arguments to mnist.py

shashank-iitbhu force-pushed the feat/hyperopt-suggestion-service-update branch from fddb763 to 282f81d Compare September 10, 2024 16:33

shashank-iitbhu commented Sep 11, 2024

View reviewed changes

shashank-iitbhu requested a review from tenzen-y September 17, 2024 12:16

hyperopt-suggestion example update

b7d09a6

Signed-off-by: Shashank Mittal <[email protected]>

tenzen-y reviewed Sep 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GSOC] `hyperopt` suggestion service logic update #2412

[GSOC] `hyperopt` suggestion service logic update #2412

shashank-iitbhu commented Aug 21, 2024

google-oss-prow bot commented Aug 21, 2024

tenzen-y commented Aug 22, 2024

andreyvelich left a comment

andreyvelich Sep 2, 2024

andreyvelich Sep 2, 2024

shashank-iitbhu Sep 11, 2024

andreyvelich Sep 2, 2024

tenzen-y Sep 3, 2024

andreyvelich Sep 3, 2024

tenzen-y Sep 3, 2024 •

edited

Loading

andreyvelich Sep 3, 2024 •

edited

Loading

tenzen-y Sep 3, 2024

andreyvelich Sep 3, 2024

tenzen-y Sep 3, 2024

shashank-iitbhu Sep 6, 2024

andreyvelich Sep 2, 2024

andreyvelich Sep 2, 2024

shashank-iitbhu Sep 11, 2024

andreyvelich Sep 2, 2024

shashank-iitbhu Sep 6, 2024 •

edited

Loading

andreyvelich Sep 6, 2024

tenzen-y Sep 19, 2024

andreyvelich Sep 2, 2024

shashank-iitbhu Sep 11, 2024 •

edited

Loading

tenzen-y Sep 19, 2024

shashank-iitbhu commented Sep 19, 2024

		@@ -533,14 +533,6 @@ func convertParameterType(typ experimentsv1beta1.ParameterType) suggestionapi.Pa

		func convertFeasibleSpace(fs experimentsv1beta1.FeasibleSpace) *suggestionapi.FeasibleSpace {
		distribution := convertDistribution(fs.Distribution)

	sigma = (float(param.max) - float(param.min)) / 6
	// We consider the normal distribution based on the range of ±3 sigma.
	sigma = (float(param.max) - float(param.min)) / 6

[GSOC] hyperopt suggestion service logic update #2412

Are you sure you want to change the base?

[GSOC] hyperopt suggestion service logic update #2412

Conversation

shashank-iitbhu commented Aug 21, 2024

google-oss-prow bot commented Aug 21, 2024

tenzen-y commented Aug 22, 2024

andreyvelich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tenzen-y Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

andreyvelich Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shashank-iitbhu Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shashank-iitbhu Sep 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shashank-iitbhu commented Sep 19, 2024

[GSOC] `hyperopt` suggestion service logic update #2412

[GSOC] `hyperopt` suggestion service logic update #2412

tenzen-y Sep 3, 2024 •

edited

Loading

andreyvelich Sep 3, 2024 •

edited

Loading

shashank-iitbhu Sep 6, 2024 •

edited

Loading

shashank-iitbhu Sep 11, 2024 •

edited

Loading