diff --git a/docs/source/index.rst b/docs/source/index.rst index 0ca5d5f3410..b132046dbf6 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -116,6 +116,7 @@ Documentation serving/sky-serve serving/service-yaml-spec + serving/autoscaling .. toctree:: :maxdepth: 1 diff --git a/docs/source/serving/autoscaling.rst b/docs/source/serving/autoscaling.rst new file mode 100644 index 00000000000..d9f2d859924 --- /dev/null +++ b/docs/source/serving/autoscaling.rst @@ -0,0 +1,121 @@ +.. _serve-autoscaling: + +Autoscaling +=========== + +SkyServe provides out-of-the-box autoscaling for your services. + +Fixed Replicas +-------------- + +In a service YAML, the number of replicas to launch is specified in the ``service`` section's ``replicas`` field: + +.. code-block:: yaml + :emphasize-lines: 3 + + service: + readiness_probe: / + replicas: 2 + + # ... + +In this case, SkyServe will launch 2 replicas of your service. However, this deployment is fixed and cannot adjust to dynamic traffic. +SkyServe provides autoscaling to help you scale your service up and down based on traffic, as shown below. + +Enabling Autoscaling +-------------------- + +Here is a minimal example to enable autoscaling for your service: + +.. code-block:: yaml + :emphasize-lines: 3-6 + + service: + readiness_probe: / + replica_policy: + min_replicas: 2 + max_replicas: 10 + target_qps_per_replica: 2.5 + + # ... + +In this example, SkyServe will: + +- Initially, launch 2 replicas of your service (``min_replicas``) +- Scale up gradually if the traffic is high, up to 10 replicas (``max_replicas``) +- Scale down gradually if the traffic is low, with a minimum of 2 replicas (``min_replicas``) + +The replica count will always be in the range +:code:`[min_replicas, max_replicas]`. + +Autoscaling is performed based on the QPS (Queries Per Second) of your service. +SkyServe will scale your service so that, ultimately, each replica receives +approximately :code:`target_qps_per_replica` queries per second. +This value can be a float; for example: + +- If the QPS is higher than 2.5 per replica, SkyServe will launch more replicas (but no more than 10 replicas) +- If the QPS is lower than 2.5 per replica, SkyServe will scale down the replicas (but no less than 2 replicas) + +Specifically, the current target number of replicas is calculated as: + +.. code-block:: python + + current_target_replicas = ceil(current_qps / target_qps_per_replica) + final_target_replicas = min(max_replicas, max(min_replicas, current_target_replicas)) + +.. tip:: + + :code:`replicas` is a shortcut for :code:`replica_policy.min_replicas`. These two fields cannot be specified at the same time. + +.. tip:: + + :code:`target_qps_per_replica` can be any positive floating point number. If processing one request takes two seconds in one replica, we can use :code:`target_qps_per_replica=0.5`. + +Scaling Delay +------------- + +SkyServe will not scale up or down immediately. Instead, SkyServe will only +upscale or downscale your service if the QPS of your service is higher or lower +than the target QPS for a period of time. This is to avoid scaling up and down +too aggressively. + +The default scaling delay is 300s for upscale and 1200s for downscale. You can +change the scaling delay by specifying the :code:`upscale_delay_seconds` and +:code:`downscale_delay_seconds` fields: + +.. code-block:: yaml + :emphasize-lines: 7-8 + + service: + readiness_probe: / + replica_policy: + min_replicas: 2 + max_replicas: 10 + target_qps_per_replica: 3 + upscale_delay_seconds: 300 + downscale_delay_seconds: 1200 + + # ... + +If you want more aggressive scaling, set those values to a lower number and vice versa. + +Scale-to-Zero +=============== + +SkyServe supports scale-to-zero. + +If your service might experience long periods of time with no traffic, consider using :code:`min_replicas: 0`: + +.. code-block:: yaml + :emphasize-lines: 4 + + service: + readiness_probe: / + replica_policy: + min_replicas: 0 + max_replicas: 3 + target_qps_per_replica: 6.3 + + # ... + +The service will scale down all replicas when there is no traffic to the system and will save costs on idle replicas. When upscaling from zero, the upscale delay will be ignored in order to bring up the service faster. diff --git a/docs/source/serving/service-yaml-spec.rst b/docs/source/serving/service-yaml-spec.rst index dd89e694ece..2a1ee29df11 100644 --- a/docs/source/serving/service-yaml-spec.rst +++ b/docs/source/serving/service-yaml-spec.rst @@ -1,7 +1,7 @@ .. _service-yaml-spec: Service YAML -========================== +============ SkyServe provides an intuitive YAML interface to specify a service. It is highly similar to the :ref:`SkyPilot task YAML `: with an additional service section in your original task YAML, you could change it to a service YAML. diff --git a/docs/source/serving/sky-serve.rst b/docs/source/serving/sky-serve.rst index fd327d3e489..50a7cd0c8ee 100644 --- a/docs/source/serving/sky-serve.rst +++ b/docs/source/serving/sky-serve.rst @@ -12,9 +12,9 @@ Why SkyServe? * **Bring any serving framework** (vLLM, TGI, FastAPI, ...) and scale it across regions/clouds * **Reduce costs and increase availability** of service replicas by leveraging multiple/cheaper locations and hardware (spot instances) -* **Out-of-the-box load-balancing and autoscaling** of service replicas +* Out-of-the-box **load-balancing** and **autoscaling** of service replicas +* **Privacy and Control**: Everything is launched inside your cloud accounts and VPCs * Manage multi-cloud, multi-region deployments with a single control plane -* **Privacy**: Everything is launched inside your cloud accounts and VPCs .. * Allocate scarce resources (e.g., A100) **across regions and clouds** .. * Autoscale your endpoint deployment with load balancing @@ -444,6 +444,12 @@ Terminate services: $ sky serve down http-server # terminate the http-server service $ sky serve down --all # terminate all services +Autoscaling +----------- + +See :ref:`Autoscaling ` for more information. + + SkyServe Architecture ---------------------