[Feature] [kubectl-plugin] Expose setting `shutdownAfterJobFinishes` and `ttlSecondsAfterFinished` in ray job submit #3627

CheyuWu · 2025-05-17T18:28:32Z

Why are these changes needed?

Currently shutdownAfterJobFinishes and ttlSecondsAfterFinished aren't being exposed in kubectl ray job submit, although they are available in the CRD.

Ray Operator

Add support for shutdownAfterJobFinishes and ttlSecondsAfterFinished in RayJob spec.
Implement validation logic: if shutdownAfterJobFinishes is False, then ttlSecondsAfterFinished must be 0 or unset.
Add corresponding unit tests.

kubectl-plugin

Add shutdownAfterJobFinishes and ttlSecondsAfterFinished flags to the rayjob submit command.
Implement validation logic: if shutdownAfterJobFinishes is False, then ttlSecondsAfterFinished must be 0 or unset.
Add corresponding unit tests.
Split TestRayJobSubmitValidate into two test cases: one for YAML-based submission and one for non-YAML-based submission.

Manual tests

Success case (--shutdown-after-job-finishes is given and --ttl-seconds-after-finished set to 10)

$ cd kubectl-plugin/test/e2e/
$ kubectl ray job submit --name rayjob-sample --runtime-env testdata/rayjob-submit-working-dir/runtime-env-sample.yaml --head-cpu 1 --head-memory 2Gi --worker-cpu 1 --worker-memory 2Gi --shutdown-after-job-finishes --ttl-seconds-after-finished 10 -- python entrypoint-python-sample.py

Failure case (--shutdown-after-job-finishes is not given)

$ cd kubectl-plugin/test/e2e/
$ kubectl ray job submit --name rayjob-sample --runtime-env testdata/rayjob-submit-working-dir/runtime-env-sample.yaml --head-cpu 1 --head-memory 2Gi --worker-cpu 1 --worker-memory 2Gi  --ttl-seconds-after-finished 10 -- python entrypoint-python-sample.py

Output

Error: ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true

Related issue number

Closes #3560

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

CheyuWu · 2025-05-17T18:33:30Z

kubectl-plugin/pkg/cmd/job/job_submit.go

+	shutdownAfterJobFinishes bool
+	ttlSecondsAfterFinished  int32


Add shutdownAfterJobFinishes and ttlSecondsAfterFinished

CheyuWu · 2025-05-17T18:34:41Z

kubectl-plugin/pkg/cmd/job/job_submit.go

+		if !options.RayJob.Spec.ShutdownAfterJobFinishes && options.RayJob.Spec.TTLSecondsAfterFinished != 0 {
+			return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")
+		}
+


Implement validation logic when --filename is provided

if shutdownAfterJobFinishes is False, then ttlSecondsAfterFinished must be 0 or unset.

CheyuWu · 2025-05-17T18:35:08Z

kubectl-plugin/pkg/cmd/job/job_submit.go

+	} else if options.fileName == "" {
+		if !options.shutdownAfterJobFinishes && options.ttlSecondsAfterFinished != 0 {
+			return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")
+		}


Implement validation logic when --filename is not provided

if shutdownAfterJobFinishes is False, then ttlSecondsAfterFinished must be 0 or unset.

CheyuWu · 2025-05-17T18:37:30Z

kubectl-plugin/pkg/cmd/job/job_submit_test.go

@@ -30,7 +30,7 @@ func TestRayJobSubmitComplete(t *testing.T) {
 	assert.Equal(t, "fake/path/to/env/yaml", fakeSubmitJobOptions.runtimeEnv)
 }

-func TestRayJobSubmitValidate(t *testing.T) {
+func TestRayJobSubmitWithYamlValidate(t *testing.T) {


Split TestRayJobSubmitValidate into two test cases: one for YAML-based submission and one for non-YAML-based submission.

CheyuWu · 2025-05-17T18:37:37Z

kubectl-plugin/pkg/cmd/job/job_submit_test.go

@@ -99,6 +122,52 @@ spec:
 	}
 }

+func TestRayJobSubmitWithoutYamlValidate(t *testing.T) {


Split TestRayJobSubmitValidate into two test cases: one for YAML-based submission and one for non-YAML-based submission.

CheyuWu · 2025-05-17T18:38:08Z

ray-operator/controllers/ray/utils/validation.go

+	if !rayJob.Spec.ShutdownAfterJobFinishes && rayJob.Spec.TTLSecondsAfterFinished != 0 {
+		return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished is non-zero")
+	}
+


Implement validation logic in ray operator

if shutdownAfterJobFinishes is False, then ttlSecondsAfterFinished must be 0 or unset.

CheyuWu · 2025-05-17T18:39:29Z

Hi @MortalHappiness and @kevin85421 PTAL

…idation and plugin

kevin85421 · 2025-05-19T17:48:49Z

cc @davidxia would you mind reviewing this PR? Thanks!

davidxia · 2025-05-20T01:20:29Z

kubectl-plugin/pkg/cmd/job/job_submit.go

@@ -254,8 +258,17 @@ func (options *SubmitJobOptions) Validate() error {
 			}
 			options.runtimeEnvJson = string(runtimeJson)
 		}
+
+		if !options.RayJob.Spec.ShutdownAfterJobFinishes && options.RayJob.Spec.TTLSecondsAfterFinished != 0 {
+			return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")


might be better to use the YAML field names ttlSecondsAfterFinished and shutdownAfterJobFinishes here

Suggested change

return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")

return fmt.Errorf("ttlSecondsAfterFinished is only supported when shutdownAfterJobFinishes is 'true'")

davidxia · 2025-05-20T02:23:17Z

kubectl-plugin/pkg/cmd/job/job_submit.go

@@ -254,8 +258,17 @@ func (options *SubmitJobOptions) Validate() error {
 			}
 			options.runtimeEnvJson = string(runtimeJson)
 		}
+
+		if !options.RayJob.Spec.ShutdownAfterJobFinishes && options.RayJob.Spec.TTLSecondsAfterFinished != 0 {


unrelated to this PR, but when I was testing this change, I ran kubectl ray job submit -f ray-job.sample.yaml --working-dir . --shutdown-after-job-finishes --ttl-seconds-after-finished 600 -- python /home/ray/samples/sample_code.py and wondered why the new flags weren't used. I had to look at this source to understand -f/--filename makes the command ignore many of these other flags.

+1. I think we should take the filename as a base, and flags should override the values in the file.

I created an issue to track this #3639

The behavior of kubectl ray create cluster is to error with an actionable error message if declarative file and CLI flags are used together. We can consider doing that here too if we think it's not worth it to have override logic.

It is worth overriding the values in the file. I will change this.

davidxia · 2025-05-20T02:38:28Z

kubectl-plugin/pkg/cmd/job/job_submit.go

 	} else if strings.TrimSpace(options.rayjobName) == "" {
 		return fmt.Errorf("Must set either yaml file (--filename) or set Ray job name (--name)")
+	} else if options.fileName == "" {
+		if !options.shutdownAfterJobFinishes && options.ttlSecondsAfterFinished != 0 {
+			return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")


Suggested change

return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")

return fmt.Errorf("--ttl-seconds-after-finished is only supported when --shutdown-after-job-finishes is set to true")

davidxia · 2025-05-20T02:39:02Z

ray-operator/controllers/ray/utils/validation.go

@@ -144,6 +144,10 @@ func ValidateRayJobSpec(rayJob *rayv1.RayJob) error {
 		return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false is not allowed to be suspended")
 	}

+	if !rayJob.Spec.ShutdownAfterJobFinishes && rayJob.Spec.TTLSecondsAfterFinished != 0 {
+		return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished is non-zero")


Suggested change

return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished is non-zero")

return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished")

davidxia · 2025-05-20T02:43:03Z

kubectl-plugin/pkg/cmd/job/job_submit.go

 	} else if strings.TrimSpace(options.rayjobName) == "" {
 		return fmt.Errorf("Must set either yaml file (--filename) or set Ray job name (--name)")
+	} else if options.fileName == "" {
+		if !options.shutdownAfterJobFinishes && options.ttlSecondsAfterFinished != 0 {


also check ttlSecondsAfterFinished ≥ 0 in both cases? Right now this command will create a RayJob with ttlSecondsAfterFinished: -10

kubectl ray --context gke_kubeflow-platform_europe-west4-b_ml-compute-1 -n hyperkube job submit --name dxia-test --shutdown-after-job-finishes --ttl-seconds-after-finished -10 --working-dir . --dry-run -- python /home/ray/samples/sample_code.py | grep -B 1 ttlSeconds submissionMode: InteractiveMode ttlSecondsAfterFinished: -10

MortalHappiness · 2025-05-20T03:24:11Z

apiserver/pkg/model/converter_test.go

-		TTLSecondsAfterFinished: secondsValue,
-		RayClusterSpec:          &ClusterSpecTest.Spec,
+		RuntimeEnvYAML:           "mytest yaml",
+		ShutdownAfterJobFinishes: true,


Why change apiserver tests?

MortalHappiness · 2025-05-20T03:25:49Z

kubectl-plugin/pkg/cmd/job/job_submit.go

@@ -171,6 +173,8 @@ func NewJobSubmitCommand(cmdFactory cmdutil.Factory, streams genericclioptions.I
 	cmd.Flags().StringVar(&options.workerGPU, "worker-gpu", "0", "number of GPUs in each worker group replica")
 	cmd.Flags().BoolVar(&options.dryRun, "dry-run", false, "print the generated YAML instead of creating the cluster. Only works when filename is not provided")
 	cmd.Flags().BoolVarP(&options.verbose, "verbose", "v", false, "Passing the '--verbose' flag to the 'ray job submit' command")
+	cmd.Flags().BoolVar(&options.shutdownAfterJobFinishes, "shutdown-after-job-finishes", false, "Shutdown the cluster after the job finishes")
+	cmd.Flags().Int32Var(&options.ttlSecondsAfterFinished, "ttl-seconds-after-finished", 0, "TTL seconds after finished. Only works when filename is not provided")


It still works when filename is provided right?

MortalHappiness · 2025-05-20T03:32:23Z

kubectl-plugin/pkg/cmd/job/job_submit.go

@@ -254,8 +258,17 @@ func (options *SubmitJobOptions) Validate() error {
 			}
 			options.runtimeEnvJson = string(runtimeJson)
 		}
+
+		if !options.RayJob.Spec.ShutdownAfterJobFinishes && options.RayJob.Spec.TTLSecondsAfterFinished != 0 {


+1. I think we should take the filename as a base, and flags should override the values in the file.

I created an issue to track this #3639

CheyuWu commented May 17, 2025

View reviewed changes

kevin85421 assigned MortalHappiness May 17, 2025

CheyuWu force-pushed the feat/jobfinished branch from ebb4f5c to 639e1d1 Compare May 18, 2025 11:36

feat: add shutdownAfterJobFinishes and ttlSecondsAfterFinished in val…

88c16de

…idation and plugin

CheyuWu force-pushed the feat/jobfinished branch from 639e1d1 to 88c16de Compare May 18, 2025 18:47

fix: api server testcase err

3d35117

CheyuWu force-pushed the feat/jobfinished branch from 28a6566 to 3d35117 Compare May 19, 2025 16:55

davidxia reviewed May 20, 2025

View reviewed changes

MortalHappiness mentioned this pull request May 20, 2025

[Feature][Kubectl-plugin] Make flags override yaml file for kubectl ray job submit #3639

Open

MortalHappiness reviewed May 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] [kubectl-plugin] Expose setting `shutdownAfterJobFinishes` and `ttlSecondsAfterFinished` in ray job submit #3627

[Feature] [kubectl-plugin] Expose setting `shutdownAfterJobFinishes` and `ttlSecondsAfterFinished` in ray job submit #3627

CheyuWu commented May 17, 2025 •

edited

Loading

CheyuWu May 17, 2025

CheyuWu May 17, 2025

CheyuWu May 17, 2025

CheyuWu May 17, 2025 •

edited

Loading

CheyuWu May 17, 2025 •

edited

Loading

CheyuWu May 17, 2025

CheyuWu commented May 17, 2025

kevin85421 commented May 19, 2025

davidxia May 20, 2025

davidxia May 20, 2025

MortalHappiness May 20, 2025 •

edited

Loading

davidxia May 20, 2025

CheyuWu May 20, 2025

davidxia May 20, 2025

davidxia May 20, 2025

davidxia May 20, 2025

MortalHappiness May 20, 2025

MortalHappiness May 20, 2025

MortalHappiness May 20, 2025 •

edited

Loading

	return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")
	return fmt.Errorf("ttlSecondsAfterFinished is only supported when shutdownAfterJobFinishes is 'true'")

	return fmt.Errorf("ttl-seconds-after-finished is only supported when shutdown-after-job-finishes is set to true")
	return fmt.Errorf("--ttl-seconds-after-finished is only supported when --shutdown-after-job-finishes is set to true")

	return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished is non-zero")
	return fmt.Errorf("a RayJob with shutdownAfterJobFinishes set to false cannot have TTLSecondsAfterFinished")

[Feature] [kubectl-plugin] Expose setting shutdownAfterJobFinishes and ttlSecondsAfterFinished in ray job submit #3627

Are you sure you want to change the base?

[Feature] [kubectl-plugin] Expose setting shutdownAfterJobFinishes and ttlSecondsAfterFinished in ray job submit #3627

Conversation

CheyuWu commented May 17, 2025 • edited Loading

Why are these changes needed?

Ray Operator

kubectl-plugin

Manual tests

Related issue number

Checks

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CheyuWu May 17, 2025 • edited Loading

Choose a reason for hiding this comment

CheyuWu May 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CheyuWu commented May 17, 2025

kevin85421 commented May 19, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MortalHappiness May 20, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MortalHappiness May 20, 2025 • edited Loading

Choose a reason for hiding this comment

[Feature] [kubectl-plugin] Expose setting `shutdownAfterJobFinishes` and `ttlSecondsAfterFinished` in ray job submit #3627

[Feature] [kubectl-plugin] Expose setting `shutdownAfterJobFinishes` and `ttlSecondsAfterFinished` in ray job submit #3627

CheyuWu commented May 17, 2025 •

edited

Loading

CheyuWu May 17, 2025 •

edited

Loading

CheyuWu May 17, 2025 •

edited

Loading

MortalHappiness May 20, 2025 •

edited

Loading

MortalHappiness May 20, 2025 •

edited

Loading