[Ray Job] : Working Dir files does not get changed even if remote URI contents get changed

[ujjawal-khare]

What happened + What you expected to happen

When submitting job to ray cluster with git/s3 URI and changing the content of URI it does not get reflected in the cluster since cleanup is not happening after the job is completed on ray cluster.

Versions / Dependencies

Ray 2.37

Reproduction script

Submit JOB to cluster with args

"runtime_env": { "pip": [ "", "smart_open" ], "working_dir": <URI> },

Change the content in URI, it will not reflected in the next job run.

Issue Severity

High: It blocks me from completing my task.

[jjyao]

@ujjawal-khare I think it's a known issue that we haven't fixed it. As a workaround, whenever you change the content, generate a new working dir URI.

[ujjawal-khare]

Hey @jjyao , I am passing github uri so it won't be possible for me to change URI. I can contribute to fix this, is there anything specific that I should be taking care of?

[ujjawal-khare-27]

@jjyao can you please help me with review for the PR: #49313