Introducing SkyServe

[cblmemo]

This PR introduces SkyServe, a lightweight multi-cloud serving system.

User doc: https://docs.google.com/document/d/1vVmzLF-EkG3Moj-q47DQBGvFipK4PNfkz0V6LyaPstE/edit#heading=h.m6tcx7d50j23

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[cblmemo]

mipl@ip-192-168-1-3 skypilot % sky serve status
E 11-14 14:21:46 subprocess_utils.py:73] Traceback (most recent call last):
E 11-14 14:21:46 subprocess_utils.py:73]   File "<string>", line 1, in <module>
E 11-14 14:21:46 subprocess_utils.py:73] AttributeError: module 'sky.serve.serve_utils' has no attribute 'get_service_status_encoded'. Did you mean: 'get_serve_status_encoded'?
E 11-14 14:21:46 subprocess_utils.py:73] 
Services
Failed to fetch service statuses due to connection issues. Please try again later. Details: [RuntimeError] Failed to fetch services
mipl@ip-192-168-1-3 skypilot % 

getting this on latest commits

This is because we made an API change. sky serve down -a -y and re-up them should fix the issue 🫡

[manishiitg]

E 11-15 09:05:20 subprocess_utils.py:73] Traceback (most recent call last):
E 11-15 09:05:20 subprocess_utils.py:73]   File "<string>", line 1, in <module>
E 11-15 09:05:20 subprocess_utils.py:73] TypeError: terminate_services() got an unexpected keyword argument 'purge'
E 11-15 09:05:20 subprocess_utils.py:73] 
sky.exceptions.CommandError: Command python3 -u -c 'from sky.serve import serve_state; from sky.serve import serve_utils; msg = serve_utils.terminate_services(None, purge=False); print(msg, end="", flush=True)' failed with return code 1.
Failed to terminate service

The above exception was the direct cause of the following exception:

RuntimeError: Failed to terminate service
mipl@ip-192-168-1-3 ~ % 

getting the same issue stil

[cblmemo]

E 11-15 09:05:20 subprocess_utils.py:73] Traceback (most recent call last):

E 11-15 09:05:20 subprocess_utils.py:73]   File "<string>", line 1, in <module>

E 11-15 09:05:20 subprocess_utils.py:73] TypeError: terminate_services() got an unexpected keyword argument 'purge'

E 11-15 09:05:20 subprocess_utils.py:73] 

sky.exceptions.CommandError: Command python3 -u -c 'from sky.serve import serve_state; from sky.serve import serve_utils; msg = serve_utils.terminate_services(None, purge=False); print(msg, end="", flush=True)' failed with return code 1.

Failed to terminate service



The above exception was the direct cause of the following exception:



RuntimeError: Failed to terminate service

mipl@ip-192-168-1-3 ~ % 

getting the same issue stil

Oh, probably try roll back to previous commit, run sky serve down and then switch to latest commit?

[manishiitg]

hi,

i went to a previous version and did sky server down -a -y and it worked.

but when i again switch to latest commits i get the same errors again?

[cblmemo]

hi,

i went to a previous version and did sky server down -a -y and it worked.

but when i again switch to latest commits i get the same errors again?

Sorry for not making it clear 🤧 You can use sky start -f <your-controller-name> to update the code in your controller. Or sky serve up any new services should work too 🫡

[Michaelvll]

Thanks for the update @cblmemo! LGTM. Just tried the branch in a new docker container and it works nicely.

[Michaelvll]

nit: a UX question: is it possible for our load balancer to notice that the user is using a curl instead of a curl -L and return a output that -L should be added?

[cblmemo]

nit: a UX question: is it possible for our load balancer to notice that the user is using a curl instead of a curl -L and return a output that -L should be added?

It seems like -L is only a client-side argument and it only affects if the curl will retry in the client if receives a redirect response. See man curl here:

       -L, --location
              (HTTP)  If  the  server  reports  that  the requested page has moved to a different location (indicated with a Location: header and a 3XX response
              code), this option will make curl redo the request on the new place. If used together with -i, --include or -I, --head, headers from all requested
              pages  will  be  shown.  When authentication is used, curl only sends its credentials to the initial host. If a redirect takes curl to a different
              host, it won't be able to intercept the user+password. See also --location-trusted on how to change this. You can limit the amount of redirects to
              follow by using the --max-redirs option.

              When  curl follows a redirect and if the request is a POST, it will do the following request with a GET if the HTTP response was 301, 302, or 303.
              If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.

              You can tell curl to not change POST requests to GET after a 30x response by using the  dedicated  options  for  that:  --post301,  --post302  and
              --post303.

              The method set with -X, --request overrides the method curl would otherwise select to use.

Also, I tried to print the fastapi.Request information w/ & w/o -L and indeed found them identical:

[manishiitg]

i got stuck due the -L issue a lot.

i was using https://github.com/huggingface/text-generation-inference to deploy by models for inference and using their official client https://github.com/huggingface/text-generation-inference/tree/main/clients/python

there client doesn't support redirects, took few hours for me to figure this out :)

this should be mentioned in documentation as a note

[cblmemo]

i got stuck due the -L issue a lot.

i was using https://github.com/huggingface/text-generation-inference to deploy by models for inference and using their official client https://github.com/huggingface/text-generation-inference/tree/main/clients/python

there client doesn't support redirects, took few hours for me to figure this out :)

this should be mentioned in documentation as a note

Thanks! We'll definitely mentioned it in the doc 🫡

[concretevitamin]

@manishiitg Do you have a preference for the default behavior of sky serve logs without any arguments?

1. print a replica's logs (e.g., the oldest / latest live replica)
2. print controller logs
3. (current behavior) don't print any logs; print hints for arguments
4. no preference
5. something else?

[manishiitg]

No preference