Deployment Issue with serving Model on RunPod

[msehsah1]

I am experiencing an issue deploying the Mixtral-8x7B-Instruct-v0.1 LLM Model on RunPod via SkyPilot. The deployment process fails during resource allocation, specifically with the port configuration. Below are the details of my deployment configuration and the error encountered

Deployment Configuration

service:
  readiness_probe: /v1/models
  replicas: 1

resources:
  ports: 8080

setup: |
  conda create -n vllm python=3.9 -y
  conda activate vllm
  pip install vllm
  pip install ray

run: |
  conda activate vllm
  python -m vllm.entrypoints.openai.api_server \
    --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
    --host 0.0.0.0 --port 8080 \
    --model mistralai/Mixtral-8x7B-Instruct-v0.1 \
    --dtype="float16"

Error Message

I 02-08 22:42:32 optimizer.py:1219] No resource satisfying <Cloud>(ports=['8080']) on RunPod.
sky.exceptions.ResourcesUnavailableError: Catalog does not contain any instances satisfying the request:
Task(run='conda activate vllm\...')
  resources: <Cloud>(ports=['8080']).

I tried to remove the port from the configuration but I got this message

Service from YAML spec: vllm.yaml
ValueError: Must only specify one port in resources. Each replica will use the port specified as application ingress port

Environment

• sky version : 1.0.0.dev20240208
• sky commit : 49bffa4

[concretevitamin]

Hello @mahmoudrasmyfathy1 -- this is because opening ports on RunPod is yet to be supported: https://github.com/skypilot-org/skypilot/blob/master/sky/clouds/runpod.py#L33-L35

Cc @cblmemo on why the error message did not surface the "not supported feature" message.

[msehsah1]

@concretevitamin , ok great thanks, so will it be supported soon ?

[cblmemo]

Hello @mahmoudrasmyfathy1 -- this is because opening ports on RunPod is yet to be supported: https://github.com/skypilot-org/skypilot/blob/master/sky/clouds/runpod.py#L33-L35

Cc @cblmemo on why the error message did not surface the "not supported feature" message.

Thanks! Fixed in #3139.

[Scott3730]

Are there any plans to support this in the future?

[Michaelvll]

Thanks for checking in @Scott3730! This is on our roadmap, and we will try to support it soon.

Related to @3323

[github-actions]

This issue is stale because it has been open 120 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[cblmemo]

This should be fixed by #3748, closing now. @msehsah1 feel free to give it a try by running pip install -U skypilot-nightly! Lemme know if you encountered other issues.