[Serve] Service update scales to zero unexpectedly?

[concretevitamin]

Repro:

• Change https://github.com/skypilot-org/skypilot/blob/master/examples/serve/http_server/task.yaml#L13 to replicas: 1
• sky serve up -n http examples/serve/http_server/task.yaml -y
• In another shell, while 1; do curl -L 34.239.227.235:30001; sleep 0.5; done
• Wait until the replica is READY and the curl's above show responses
• Change the YAML again to

  # replicas: 1
  replica_policy:
    min_replicas: 0
    max_replicas: 1

• sky serve update http examples/serve/http_server/task.yaml -y

Observed: Status immediately became

Services
NAME  VERSION  UPTIME  STATUS      REPLICAS  ENDPOINT
http  -        6m 48s  NO_REPLICA  0/1       34.239.227.235:30001

Service Replicas
SERVICE_NAME  ID  VERSION  IP              LAUNCHED    RESOURCES       STATUS         REGION
http          1   1        44.213.109.175  7 mins ago  1x AWS(vCPU=2)  SHUTTING_DOWN  us-east-1

After a while it became

Services
NAME  VERSION  UPTIME  STATUS      REPLICAS  ENDPOINT
http  -        8m 14s  NO_REPLICA  0/0       34.239.227.235:30001

Service Replicas
No existing replicas.

The curl's are still ongoing and fail to wake up the service.

Both problems seem unexpected. Since there's constant traffic hitting the service, scale-to-zero shouldn't have kicked in. Also, why is the service not woken up?

commit 82c50f5

cc @cblmemo @Michaelvll

[cblmemo]

Oh I suppose that is because you didn't set target_qps_per_replica... Reference here

skypilot/sky/serve/autoscalers.py

Lines 167 to 171 in 82c50f5

	def _cal_target_num_replicas_based_on_qps(self) -> int:
	# Recalculate target_num_replicas based on QPS.
	# Reclip self.target_num_replicas with new min and max replicas.
	if self.target_qps_per_replica is None:
	return self.min_replicas

The UX in current master is problematic, i.e. showing Autoscaling from 0 to 1 replica when autoscaling is not enabled. We need to prompt the user if target_qps_per_replica is not set, the autoscaling will not enabled. Just submitted a PR to fix this #3361