Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ray Job] : Working Dir files does not get changed even if remote URI contents get changed #49036

Closed
ujjawal-khare opened this issue Dec 3, 2024 · 3 comments · Fixed by #49313
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core core-runtime-env Issues related to Ray environment dependencies P1 Issue that should be fixed within a few weeks

Comments

@ujjawal-khare
Copy link
Contributor

What happened + What you expected to happen

When submitting job to ray cluster with git/s3 URI and changing the content of URI it does not get reflected in the cluster since cleanup is not happening after the job is completed on ray cluster.

Versions / Dependencies

Ray 2.37

Reproduction script

Submit JOB to cluster with args

"runtime_env": { "pip": [ "", "smart_open" ], "working_dir": <URI> },

Change the content in URI, it will not reflected in the next job run.

Issue Severity

High: It blocks me from completing my task.

@ujjawal-khare ujjawal-khare added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 3, 2024
@jcotant1 jcotant1 added the core Issues that should be addressed in Ray Core label Dec 3, 2024
@jjyao
Copy link
Collaborator

jjyao commented Dec 9, 2024

@ujjawal-khare I think it's a known issue that we haven't fixed it. As a workaround, whenever you change the content, generate a new working dir URI.

@jjyao jjyao added core-runtime-env Issues related to Ray environment dependencies P1 Issue that should be fixed within a few weeks and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 9, 2024
@ujjawal-khare
Copy link
Contributor Author

Hey @jjyao , I am passing github uri so it won't be possible for me to change URI. I can contribute to fix this, is there anything specific that I should be taking care of?

@ujjawal-khare-27
Copy link
Contributor

@jjyao can you please help me with review for the PR: #49313

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core core-runtime-env Issues related to Ray environment dependencies P1 Issue that should be fixed within a few weeks
Projects
None yet
4 participants