Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serve] cannot down the service with purge. #2566

Closed
MaoZiming opened this issue Sep 17, 2023 · 3 comments
Closed

[Serve] cannot down the service with purge. #2566

MaoZiming opened this issue Sep 17, 2023 · 3 comments

Comments

@MaoZiming
Copy link
Collaborator

MaoZiming commented Sep 17, 2023

I have a service that is stuck at "shutting down"

(sky) {23:18}~/Documents/skypilot:serve  ➭ sky serve status
Services
NAME          UPTIME  STATUS           REPLICAS  ENDPOINT  
service-7f28  -       SHUTTING_DOWN    0/0       - 

When I try sky serve down service-7f28 --purge, it says

Tearing down 1 service: service-7f28. Proceed? [Y/n]: Y
Tearing down 1 service ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--W 09-16 23:28:28 execution.py:1272] Ignoring error when cleaning replicas of service-7f28: Command python3 -u -c 'from sky.serve import serve_utils; msg = serve_utils.terminate_service(20002); print(msg, end="", flush=True)' failed with return code 1.
W 09-16 23:28:28 execution.py:1272] Failed when submit terminate request to controller of service service-7f28
Tearing down 1 service ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% -:--:--E 09-16 23:28:38 storage.py:762] Could not get StoreType.GCS store with name skypilot-workdir-maoziming-d78055d2.
google.api_core.exceptions.NotFound: 404 GET https://storage.googleapis.com/storage/v1/b/skypilot-workdir-maoziming-d78055d2?projection=noAcl&prettyPrint=false: The specified bucket does not exist.

The above exception was the direct cause of the following exception:

sky.exceptions.StorageBucketGetError: Attempted to connect to a non-existent bucket: gs://skypilot-workdir-maoziming-d78055d2

I don't recall deleting that bucket. I couldn't find a way to clear that service.

@MaoZiming MaoZiming changed the title [Azure / Serve] cannot down the service with purge. [Serve] cannot down the service with purge. Sep 17, 2023
@Michaelvll
Copy link
Collaborator

Is this issue already fixed in #2458 @cblmemo ?

@MaoZiming
Copy link
Collaborator Author

yes

@cblmemo
Copy link
Collaborator

cblmemo commented Nov 16, 2023

Yes it is fixed 🫡

@cblmemo cblmemo closed this as completed Nov 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants