I want ray jobs created by users to be automatically cleaned up, rather than be left around. This can be set if the user were to create and manage their own CRD yaml files, but not when using the plugin CLI.

Hi @ashrielbrian, thank you for your feedback. Would you mind sharing more details about how you use kubectl ray job submit? Is the person who creates the RayJob YAML the same as the one who runs the kubectl ray job submit command? Or do the infra/platform teams create the RayJob YAML while MLEs or data scientists run the CLI command?

also cc @MortalHappiness @davidxia for input

@CheyuWu are you interested in taking this issue if this request makes sense to Chi-Sheng and David?

Hi @kevin85421 , sure, no problem

Makes sense to me

I want ray jobs created by users to be automatically cleaned up, rather than be left around. This can be set if the user were to create and manage their own CRD yaml files, but not when using the plugin CLI.

Hi @ashrielbrian, thank you for your feedback. Would you mind sharing more details about how you use kubectl ray job submit? Is the person who creates the RayJob YAML the same as the one who runs the kubectl ray job submit command? Or do the infra/platform teams create the RayJob YAML while MLEs or data scientists run the CLI command?

Let's wait for a response on this. As for me, I don't think we should support all the fields that a regular RayJob has in the kubectl ray job submit command, because the kubectl plugin is designed primarily for data scientists, not for the infrastructure team. We've already supported reading from a YAML file in the kubectl ray job submit command, so technically all fields can be configured through that. Therefore, unless it's absolutely necessary for data scientists, we won't add more flags to support additional RayJob fields.

@MortalHappiness this feature requests make sense to me if the following two assumptions are correct:

• Data scientists should not touch the RayJob CR YAML.
• Infra/Platform teams don't own the workloads, and different workloads may have different requirements for shutdownAfterJobFinishes and ttlSecondsAfterFinished.

@davidxia has relevant infra/platform team experience. Would you mind sharing your thoughts on these two assumptions?

yes, in my team both the assumptions are the case for most users. Sometimes we make users use their own CR YAML when it's an infrequently requested config, but when we think many users might use them, we make them into CLI flags for our internal tools.

@davidxia Thank you! @MortalHappiness does this make sense to you?

Hi @kevin85421,

Originally, our data scientists had to craft their own RayJob CR yaml files, but this very quickly became a painful UX for them, and we also wanted to remove the need to do kubectl apply across any kubernetes resource, CRDs or otherwise. We (platform) started experimenting with the plugin and then integrated that with our internal CLI tool as a light wrapper around kubectl ray job submit .... So our current flow now discourages any direct editing of RayJob CRDs for data scientists - using the CLI is the primary and recommended way in our workflow to run ray jobs for our DS users.

An alternative we are considering is having our platform team manage a templated CRD, and our internal tool exposes CLI flags for specific parameters we'd like to expose, but this would do away with using the plugin as a wrapper.

Thanks @ashrielbrian! This feature request makes sense to me. @CheyuWu you can start to work on it.

@davidxia Thank you! @MortalHappiness does this make sense to you?

Yes, this makes sense to me.