vLLM tutorial doesn't work (cannot find vllm module)

[WesleyYue]

Bug

• I'm following this guide.
• It doesn't work out of the box. Specifically, it seems to be unable to see the installed packages during setup (including vllm) /home/azureuser/skypilot-runtime/bin/python: Error while finding module specification for 'vllm.entrypoints.openai.api_server' (ModuleNotFoundError: No module named 'vllm')
• I have SSH'd into the machine to check that manually doing conda activate qwen && pip list shows vllm as expected. So it only is not visible during the run steps for some reason.

To reproduce

1. Run sky launch -c qwen skypilot.yaml --cloud azure --region westus3
2. See that vllm server doesn't come up because it can't find the vllm python module
   • /home/azureuser/skypilot-runtime/bin/python: Error while finding module specification for 'vllm.entrypoints.openai.api_server' (ModuleNotFoundError: No module named 'vllm')

skypilot.yaml (modifed from qwen-7b.yaml, with extra logging statements)

envs:
  MODEL_NAME: Qwen/Qwen1.5-7B-Chat

service:
  # Specifying the path to the endpoint to check the readiness of the replicas.
  readiness_probe:
    path: /v1/chat/completions
    post_data:
      model: $MODEL_NAME
      messages:
        - role: user
          content: Hello! What is your name?
      max_tokens: 1
    initial_delay_seconds: 1200
  # How many replicas to manage.
  replicas: 1

resources:
  # accelerators: { L4, A10g, A10, L40, A40, A100, A100-80GB }
  accelerators: { A100-80GB:1 }
  disk_tier: best
  ports: 8000

setup: |
  echo "[skypilot.yaml] Activating conda environment 'qwen'"
  conda activate qwen
  if [ $? -ne 0 ]; then
    echo "[skypilot.yaml] Creating new conda environment 'qwen' with Python 3.10"
    conda create -n qwen python=3.10 -y
    conda activate qwen
  fi
  echo "[skypilot.yaml] Installing required packages..."
  pip install -U vllm==0.3.2
  pip install -U transformers==4.38.0
  echo "[skypilot.yaml] Done installing packages."

run: |
  echo "[skypilot.yaml] Listing available conda environments:"
  conda env list
  echo "[skypilot.yaml] Activating conda environment 'qwen'"
  conda activate qwen
  echo "[skypilot.yaml] Listing available conda environments:"
  conda env list
  echo "[skypilot.yaml] Listing installed packages:"
  pip list
  echo "[skypilot.yaml] Setting PATH to include /sbin"
  export PATH=$PATH:/sbin
  echo "[skypilot.yaml] Starting vllm OpenAI API server with the following configuration:"
  echo "[skypilot.yaml]   - Host: 0.0.0.0"
  echo "[skypilot.yaml]   - Model: $MODEL_NAME"
  echo "[skypilot.yaml]   - Tensor Parallel Size: $SKYPILOT_NUM_GPUS_PER_NODE"
  echo "[skypilot.yaml]   - Maximum Model Length: 1024"
  python -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 \
    --model $MODEL_NAME \
    --tensor-parallel-size $SKYPILOT_NUM_GPUS_PER_NODE \
    --max-model-len 1024 | tee ~/openai_api_server.log

Version & Commit info:

• sky -v: skypilot, version 1.0.0-dev2024053101
• sky -c: skypilot, commit e620ccc418ee69d70e580a703569d7cf0508b0ce

[WesleyYue]

Full logs here:

Task from YAML spec: x.yaml
I 06-05 10:38:15 cli.py:1112] Service section will be ignored when using `sky launch`. 
I 06-05 10:38:15 cli.py:1112] To spin up a service, use SkyServe CLI: sky serve up
I 06-05 10:38:15 optimizer.py:695] == Optimizer ==
I 06-05 10:38:15 optimizer.py:718] Estimated cost: $3.7 / hour
I 06-05 10:38:15 optimizer.py:718] 
I 06-05 10:38:15 optimizer.py:843] Considered resources (1 node):
I 06-05 10:38:15 optimizer.py:913] -------------------------------------------------------------------------------------------------------
I 06-05 10:38:15 optimizer.py:913]  CLOUD   INSTANCE                   vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE   COST ($)   CHOSEN   
I 06-05 10:38:15 optimizer.py:913] -------------------------------------------------------------------------------------------------------
I 06-05 10:38:15 optimizer.py:913]  Azure   Standard_NC24ads_A100_v4   24      220       A100-80GB:1    westus3       3.67          ✔     
I 06-05 10:38:15 optimizer.py:913] -------------------------------------------------------------------------------------------------------
I 06-05 10:38:15 optimizer.py:913] 
Launching a new cluster 'qwen'. Proceed? [Y/n]: y
I 06-05 10:38:20 cloud_vm_ray_backend.py:4397] Creating a new cluster: 'qwen' [1x Azure(Standard_NC24ads_A100_v4, {'A100-80GB': 1}, disk_tier=best, ports=['8000'])].
I 06-05 10:38:20 cloud_vm_ray_backend.py:4397] Tip: to reuse an existing cluster, specify --cluster (-c). Run `sky status` to see existing clusters.
I 06-05 10:38:20 cloud_vm_ray_backend.py:1385] To view detailed progress: tail -n100 -f /Users/wesley/sky_logs/sky-2024-06-05-10-38-15-867861/provision.log
I 06-05 10:38:20 cloud_vm_ray_backend.py:1779] Launching on Azure westus3
I 06-05 10:42:50 log_utils.py:45] Head node is up.
I 06-05 10:52:47 cloud_vm_ray_backend.py:1627] Successfully provisioned or found existing VM.
I 06-05 10:52:51 cloud_vm_ray_backend.py:3215] Running setup on 1 node.
[skypilot.yaml] Activating conda environment 'qwen'

EnvironmentNameNotFound: Could not find conda environment: qwen
You can list all discoverable environments with `conda info --envs`.


[skypilot.yaml] Creating new conda environment 'qwen' with Python 3.10
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/azureuser/miniconda3/envs/qwen

  added / updated specs:
    - python=3.10


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bzip2-1.0.8                |       h5eee18b_6         262 KB
    ca-certificates-2024.3.11  |       h06a4308_0         127 KB
    libffi-3.4.4               |       h6a678d5_1         141 KB
    openssl-3.0.13             |       h7f8727e_2         5.2 MB
    pip-24.0                   |  py310h06a4308_0         2.7 MB
    python-3.10.14             |       h955ad1f_1        26.8 MB
    setuptools-69.5.1          |  py310h06a4308_0        1012 KB
    sqlite-3.45.3              |       h5eee18b_0         1.2 MB
    tk-8.6.14                  |       h39e8969_0         3.4 MB
    tzdata-2024a               |       h04d1e81_0         116 KB
    wheel-0.43.0               |  py310h06a4308_0         110 KB
    xz-5.4.6                   |       h5eee18b_1         643 KB
    zlib-1.2.13                |       h5eee18b_1         111 KB
    ------------------------------------------------------------
                                           Total:        41.8 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main 
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu 
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6 
  ca-certificates    pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0 
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 
  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_1 
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 
  libgomp            pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 
  libuuid            pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0 
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 
  openssl            pkgs/main/linux-64::openssl-3.0.13-h7f8727e_2 
  pip                pkgs/main/linux-64::pip-24.0-py310h06a4308_0 
  python             pkgs/main/linux-64::python-3.10.14-h955ad1f_1 
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0 
  setuptools         pkgs/main/linux-64::setuptools-69.5.1-py310h06a4308_0 
  sqlite             pkgs/main/linux-64::sqlite-3.45.3-h5eee18b_0 
  tk                 pkgs/main/linux-64::tk-8.6.14-h39e8969_0 
  tzdata             pkgs/main/noarch::tzdata-2024a-h04d1e81_0 
  wheel              pkgs/main/linux-64::wheel-0.43.0-py310h06a4308_0 
  xz                 pkgs/main/linux-64::xz-5.4.6-h5eee18b_1 
  zlib               pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1 



Downloading and Extracting Packages: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate qwen
#
# To deactivate an active environment, use
#
#     $ conda deactivate

[skypilot.yaml] Installing required packages...
Collecting vllm==0.3.2
  Downloading vllm-0.3.2-cp310-cp310-manylinux1_x86_64.whl.metadata (7.5 kB)
Collecting ninja (from vllm==0.3.2)
  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)
Collecting psutil (from vllm==0.3.2)
  Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting ray>=2.9 (from vllm==0.3.2)
  Downloading ray-2.23.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (13 kB)
Collecting sentencepiece (from vllm==0.3.2)
  Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)
Collecting numpy (from vllm==0.3.2)
  Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 2.4 MB/s eta 0:00:00
Collecting torch==2.1.2 (from vllm==0.3.2)
  Downloading torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata (25 kB)
Collecting transformers>=4.38.0 (from vllm==0.3.2)
  Downloading transformers-4.41.2-py3-none-any.whl.metadata (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.8/43.8 kB 1.9 MB/s eta 0:00:00
Collecting xformers==0.0.23.post1 (from vllm==0.3.2)
  Downloading xformers-0.0.23.post1-cp310-cp310-manylinux2014_x86_64.whl.metadata (1.0 kB)
Collecting fastapi (from vllm==0.3.2)
  Downloading fastapi-0.111.0-py3-none-any.whl.metadata (25 kB)
Collecting uvicorn[standard] (from vllm==0.3.2)
  Downloading uvicorn-0.30.1-py3-none-any.whl.metadata (6.3 kB)
Collecting pydantic>=2.0 (from vllm==0.3.2)
  Downloading pydantic-2.7.3-py3-none-any.whl.metadata (108 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.0/109.0 kB 7.9 MB/s eta 0:00:00
Collecting aioprometheus[starlette] (from vllm==0.3.2)
  Downloading aioprometheus-23.12.0-py3-none-any.whl.metadata (9.8 kB)
Collecting pynvml==11.5.0 (from vllm==0.3.2)
  Downloading pynvml-11.5.0-py3-none-any.whl.metadata (7.8 kB)
Collecting triton>=2.1.0 (from vllm==0.3.2)
  Downloading triton-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.4 kB)
Collecting cupy-cuda12x==12.1.0 (from vllm==0.3.2)
  Downloading cupy_cuda12x-12.1.0-cp310-cp310-manylinux2014_x86_64.whl.metadata (2.6 kB)
Collecting fastrlock>=0.5 (from cupy-cuda12x==12.1.0->vllm==0.3.2)
  Downloading fastrlock-0.8.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl.metadata (9.3 kB)
Collecting filelock (from torch==2.1.2->vllm==0.3.2)
  Downloading filelock-3.14.0-py3-none-any.whl.metadata (2.8 kB)
Collecting typing-extensions (from torch==2.1.2->vllm==0.3.2)
  Downloading typing_extensions-4.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting sympy (from torch==2.1.2->vllm==0.3.2)
  Downloading sympy-1.12.1-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch==2.1.2->vllm==0.3.2)
  Downloading networkx-3.3-py3-none-any.whl.metadata (5.1 kB)
Collecting jinja2 (from torch==2.1.2->vllm==0.3.2)
  Downloading jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting fsspec (from torch==2.1.2->vllm==0.3.2)
  Downloading fsspec-2024.6.0-py3-none-any.whl.metadata (11 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-cupti-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu12==8.9.2.26 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cublas-cu12==12.1.3.1 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cufft-cu12==11.0.2.54 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu12==10.3.2.106 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cusolver-cu12==11.4.5.107 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu12==12.1.0.106 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu12==2.18.1 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu12==12.1.105 (from torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting triton>=2.1.0 (from vllm==0.3.2)
  Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)
Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch==2.1.2->vllm==0.3.2)
  Downloading nvidia_nvjitlink_cu12-12.5.40-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting annotated-types>=0.4.0 (from pydantic>=2.0->vllm==0.3.2)
  Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.18.4 (from pydantic>=2.0->vllm==0.3.2)
  Downloading pydantic_core-2.18.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.5 kB)
Collecting click>=7.0 (from ray>=2.9->vllm==0.3.2)
  Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting jsonschema (from ray>=2.9->vllm==0.3.2)
  Downloading jsonschema-4.22.0-py3-none-any.whl.metadata (8.2 kB)
Collecting msgpack<2.0.0,>=1.0.0 (from ray>=2.9->vllm==0.3.2)
  Downloading msgpack-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.1 kB)
Collecting packaging (from ray>=2.9->vllm==0.3.2)
  Downloading packaging-24.0-py3-none-any.whl.metadata (3.2 kB)
Collecting protobuf!=3.19.5,>=3.15.3 (from ray>=2.9->vllm==0.3.2)
  Downloading protobuf-5.27.0-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Collecting pyyaml (from ray>=2.9->vllm==0.3.2)
  Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting aiosignal (from ray>=2.9->vllm==0.3.2)
  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)
Collecting frozenlist (from ray>=2.9->vllm==0.3.2)
  Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)
Collecting requests (from ray>=2.9->vllm==0.3.2)
  Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB)
Collecting huggingface-hub<1.0,>=0.23.0 (from transformers>=4.38.0->vllm==0.3.2)
  Downloading huggingface_hub-0.23.3-py3-none-any.whl.metadata (12 kB)
Collecting regex!=2019.12.17 (from transformers>=4.38.0->vllm==0.3.2)
  Downloading regex-2024.5.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.9/40.9 kB 2.2 MB/s eta 0:00:00
Collecting tokenizers<0.20,>=0.19 (from transformers>=4.38.0->vllm==0.3.2)
  Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Collecting safetensors>=0.4.1 (from transformers>=4.38.0->vllm==0.3.2)
  Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting tqdm>=4.27 (from transformers>=4.38.0->vllm==0.3.2)
  Downloading tqdm-4.66.4-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 4.3 MB/s eta 0:00:00
Collecting orjson (from aioprometheus[starlette]->vllm==0.3.2)
  Downloading orjson-3.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.7/49.7 kB 3.8 MB/s eta 0:00:00
Collecting quantile-python>=1.1 (from aioprometheus[starlette]->vllm==0.3.2)
  Downloading quantile-python-1.1.tar.gz (2.9 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting starlette>=0.14.2 (from aioprometheus[starlette]->vllm==0.3.2)
  Downloading starlette-0.37.2-py3-none-any.whl.metadata (5.9 kB)
Collecting fastapi-cli>=0.0.2 (from fastapi->vllm==0.3.2)
  Downloading fastapi_cli-0.0.4-py3-none-any.whl.metadata (7.0 kB)
Collecting httpx>=0.23.0 (from fastapi->vllm==0.3.2)
  Downloading httpx-0.27.0-py3-none-any.whl.metadata (7.2 kB)
Collecting python-multipart>=0.0.7 (from fastapi->vllm==0.3.2)
  Downloading python_multipart-0.0.9-py3-none-any.whl.metadata (2.5 kB)
Collecting ujson!=4.0.2,!=4.1.0,!=4.2.0,!=4.3.0,!=5.0.0,!=5.1.0,>=4.0.1 (from fastapi->vllm==0.3.2)
  Downloading ujson-5.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.3 kB)
Collecting email_validator>=2.0.0 (from fastapi->vllm==0.3.2)
  Downloading email_validator-2.1.1-py3-none-any.whl.metadata (26 kB)
Collecting h11>=0.8 (from uvicorn[standard]->vllm==0.3.2)
  Downloading h11-0.14.0-py3-none-any.whl.metadata (8.2 kB)
Collecting httptools>=0.5.0 (from uvicorn[standard]->vllm==0.3.2)
  Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.6 kB)
Collecting python-dotenv>=0.13 (from uvicorn[standard]->vllm==0.3.2)
  Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]->vllm==0.3.2)
  Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting watchfiles>=0.13 (from uvicorn[standard]->vllm==0.3.2)
  Downloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.9 kB)
Collecting websockets>=10.4 (from uvicorn[standard]->vllm==0.3.2)
  Downloading websockets-12.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting dnspython>=2.0.0 (from email_validator>=2.0.0->fastapi->vllm==0.3.2)
  Downloading dnspython-2.6.1-py3-none-any.whl.metadata (5.8 kB)
Collecting idna>=2.0.0 (from email_validator>=2.0.0->fastapi->vllm==0.3.2)
  Downloading idna-3.7-py3-none-any.whl.metadata (9.9 kB)
Collecting typer>=0.12.3 (from fastapi-cli>=0.0.2->fastapi->vllm==0.3.2)
  Downloading typer-0.12.3-py3-none-any.whl.metadata (15 kB)
Collecting anyio (from httpx>=0.23.0->fastapi->vllm==0.3.2)
  Downloading anyio-4.4.0-py3-none-any.whl.metadata (4.6 kB)
Collecting certifi (from httpx>=0.23.0->fastapi->vllm==0.3.2)
  Downloading certifi-2024.6.2-py3-none-any.whl.metadata (2.2 kB)
Collecting httpcore==1.* (from httpx>=0.23.0->fastapi->vllm==0.3.2)
  Downloading httpcore-1.0.5-py3-none-any.whl.metadata (20 kB)
Collecting sniffio (from httpx>=0.23.0->fastapi->vllm==0.3.2)
  Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch==2.1.2->vllm==0.3.2)
  Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting attrs>=22.2.0 (from jsonschema->ray>=2.9->vllm==0.3.2)
  Downloading attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
Collecting jsonschema-specifications>=2023.03.6 (from jsonschema->ray>=2.9->vllm==0.3.2)
  Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
Collecting referencing>=0.28.4 (from jsonschema->ray>=2.9->vllm==0.3.2)
  Downloading referencing-0.35.1-py3-none-any.whl.metadata (2.8 kB)
Collecting rpds-py>=0.7.1 (from jsonschema->ray>=2.9->vllm==0.3.2)
  Downloading rpds_py-0.18.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting charset-normalizer<4,>=2 (from requests->ray>=2.9->vllm==0.3.2)
  Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB)
Collecting urllib3<3,>=1.21.1 (from requests->ray>=2.9->vllm==0.3.2)
  Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting mpmath<1.4.0,>=1.1.0 (from sympy->torch==2.1.2->vllm==0.3.2)
  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting exceptiongroup>=1.0.2 (from anyio->httpx>=0.23.0->fastapi->vllm==0.3.2)
  Downloading exceptiongroup-1.2.1-py3-none-any.whl.metadata (6.6 kB)
Collecting shellingham>=1.3.0 (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.3.2)
  Downloading shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting rich>=10.11.0 (from typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.3.2)
  Downloading rich-13.7.1-py3-none-any.whl.metadata (18 kB)
Collecting markdown-it-py>=2.2.0 (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.3.2)
  Downloading markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB)
Collecting pygments<3.0.0,>=2.13.0 (from rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.3.2)
  Downloading pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=10.11.0->typer>=0.12.3->fastapi-cli>=0.0.2->fastapi->vllm==0.3.2)
  Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Downloading vllm-0.3.2-cp310-cp310-manylinux1_x86_64.whl (41.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.4/41.4 MB 23.4 MB/s eta 0:00:00
Downloading cupy_cuda12x-12.1.0-cp310-cp310-manylinux2014_x86_64.whl (83.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.0/83.0 MB 11.8 MB/s eta 0:00:00
Downloading pynvml-11.5.0-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 kB 4.2 MB/s eta 0:00:00
Downloading torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 670.2/670.2 MB 2.1 MB/s eta 0:00:00
Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.2/89.2 MB 9.9 MB/s eta 0:00:00
Downloading xformers-0.0.23.post1-cp310-cp310-manylinux2014_x86_64.whl (213.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 213.0/213.0 MB 5.2 MB/s eta 0:00:00
Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 410.6/410.6 MB 3.2 MB/s eta 0:00:00
Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 9.7 MB/s eta 0:00:00
Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23.7/23.7 MB 10.8 MB/s eta 0:00:00
Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 823.6/823.6 kB 38.9 MB/s eta 0:00:00
Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 731.7/731.7 MB 1.2 MB/s eta 0:00:00
Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.6/121.6 MB 1.6 MB/s eta 0:00:00
Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.5/56.5 MB 13.1 MB/s eta 0:00:00
Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.2/124.2 MB 2.0 MB/s eta 0:00:00
Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.0/196.0 MB 6.4 MB/s eta 0:00:00
Downloading nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.8 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 209.8/209.8 MB 4.4 MB/s eta 0:00:00
Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.1/99.1 kB 15.9 kB/s eta 0:00:00
Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 47.7 MB/s eta 0:00:00
Downloading pydantic-2.7.3-py3-none-any.whl (409 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 409.6/409.6 kB 28.3 MB/s eta 0:00:00
Downloading pydantic_core-2.18.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 68.8 MB/s eta 0:00:00
Downloading ray-2.23.0-cp310-cp310-manylinux2014_x86_64.whl (65.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.7/65.7 MB 12.1 MB/s eta 0:00:00
Downloading transformers-4.41.2-py3-none-any.whl (9.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.1/9.1 MB 78.2 MB/s eta 0:00:00
Downloading fastapi-0.111.0-py3-none-any.whl (91 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.0/92.0 kB 6.5 MB/s eta 0:00:00
Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.2/307.2 kB 20.2 MB/s eta 0:00:00
Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 288.2/288.2 kB 19.6 MB/s eta 0:00:00
Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.3/1.3 MB 68.2 MB/s eta 0:00:00
Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Downloading click-8.1.7-py3-none-any.whl (97 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 7.9 MB/s eta 0:00:00
Downloading email_validator-2.1.1-py3-none-any.whl (30 kB)
Downloading fastapi_cli-0.0.4-py3-none-any.whl (9.5 kB)
Downloading fastrlock-0.8.2-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_28_x86_64.whl (51 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.3/51.3 kB 3.9 MB/s eta 0:00:00
Downloading h11-0.14.0-py3-none-any.whl (58 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 4.8 MB/s eta 0:00:00
Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.4/341.4 kB 19.1 MB/s eta 0:00:00
Downloading httpx-0.27.0-py3-none-any.whl (75 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.6/75.6 kB 5.7 MB/s eta 0:00:00
Downloading httpcore-1.0.5-py3-none-any.whl (77 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.9/77.9 kB 5.7 MB/s eta 0:00:00
Downloading huggingface_hub-0.23.3-py3-none-any.whl (401 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 401.7/401.7 kB 25.5 MB/s eta 0:00:00
Downloading fsspec-2024.6.0-py3-none-any.whl (176 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 176.9/176.9 kB 12.5 MB/s eta 0:00:00
Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.3/133.3 kB 10.1 MB/s eta 0:00:00
Downloading msgpack-1.0.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (385 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 385.1/385.1 kB 25.5 MB/s eta 0:00:00
Downloading orjson-3.10.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.5/142.5 kB 11.8 MB/s eta 0:00:00
Downloading packaging-24.0-py3-none-any.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.5/53.5 kB 3.9 MB/s eta 0:00:00
Downloading protobuf-5.27.0-cp38-abi3-manylinux2014_x86_64.whl (309 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 309.2/309.2 kB 23.4 MB/s eta 0:00:00
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading python_multipart-0.0.9-py3-none-any.whl (22 kB)
Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 kB 21.3 MB/s eta 0:00:00
Downloading regex-2024.5.15-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (775 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 775.1/775.1 kB 45.1 MB/s eta 0:00:00
Downloading safetensors-0.4.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 65.5 MB/s eta 0:00:00
Downloading starlette-0.37.2-py3-none-any.whl (71 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.9/71.9 kB 5.8 MB/s eta 0:00:00
Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 96.3 MB/s eta 0:00:00
Downloading tqdm-4.66.4-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.3/78.3 kB 6.1 MB/s eta 0:00:00
Downloading typing_extensions-4.12.1-py3-none-any.whl (37 kB)
Downloading ujson-5.10.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (53 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.6/53.6 kB 4.0 MB/s eta 0:00:00
Downloading uvicorn-0.30.1-py3-none-any.whl (62 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.4/62.4 kB 5.3 MB/s eta 0:00:00
Downloading uvloop-0.19.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 97.5 MB/s eta 0:00:00
Downloading watchfiles-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 56.1 MB/s eta 0:00:00
Downloading websockets-12.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (130 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 130.2/130.2 kB 10.0 MB/s eta 0:00:00
Downloading aioprometheus-23.12.0-py3-none-any.whl (31 kB)
Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.5/239.5 kB 17.2 MB/s eta 0:00:00
Downloading filelock-3.14.0-py3-none-any.whl (12 kB)
Downloading jsonschema-4.22.0-py3-none-any.whl (88 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.3/88.3 kB 7.3 MB/s eta 0:00:00
Downloading networkx-3.3-py3-none-any.whl (1.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.7/1.7 MB 67.6 MB/s eta 0:00:00
Downloading requests-2.32.3-py3-none-any.whl (64 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 4.8 MB/s eta 0:00:00
Downloading sympy-1.12.1-py3-none-any.whl (5.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 105.2 MB/s eta 0:00:00
Downloading anyio-4.4.0-py3-none-any.whl (86 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.8/86.8 kB 6.2 MB/s eta 0:00:00
Downloading attrs-23.2.0-py3-none-any.whl (60 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 4.8 MB/s eta 0:00:00
Downloading certifi-2024.6.2-py3-none-any.whl (164 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.4/164.4 kB 11.9 MB/s eta 0:00:00
Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.1/142.1 kB 11.0 MB/s eta 0:00:00
Downloading dnspython-2.6.1-py3-none-any.whl (307 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.7/307.7 kB 18.7 MB/s eta 0:00:00
Downloading idna-3.7-py3-none-any.whl (66 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 kB 4.5 MB/s eta 0:00:00
Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 37.8 MB/s eta 0:00:00
Downloading referencing-0.35.1-py3-none-any.whl (26 kB)
Downloading rpds_py-0.18.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 61.6 MB/s eta 0:00:00
Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
Downloading typer-0.12.3-py3-none-any.whl (47 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.2/47.2 kB 4.0 MB/s eta 0:00:00
Downloading urllib3-2.2.1-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 9.7 MB/s eta 0:00:00
Downloading nvidia_nvjitlink_cu12-12.5.40-py3-none-manylinux2014_x86_64.whl (21.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.3/21.3 MB 47.0 MB/s eta 0:00:00
Downloading exceptiongroup-1.2.1-py3-none-any.whl (16 kB)
Downloading rich-13.7.1-py3-none-any.whl (240 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.7/240.7 kB 18.1 MB/s eta 0:00:00
Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 6.9 MB/s eta 0:00:00
Downloading pygments-2.18.0-py3-none-any.whl (1.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 48.0 MB/s eta 0:00:00
Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Building wheels for collected packages: quantile-python
  Building wheel for quantile-python (setup.py): started
  Building wheel for quantile-python (setup.py): finished with status 'done'
  Created wheel for quantile-python: filename=quantile_python-1.1-py3-none-any.whl size=3443 sha256=ffad0de804375a07aefad916d3681a4cd75f0599f55bec832bfa1072e98c1786
  Stored in directory: /home/azureuser/.cache/pip/wheels/6d/f4/0a/0e7d01548a005f9f3fa23101f071d248da052f2a9bf2fe11c6
Successfully built quantile-python
Installing collected packages: sentencepiece, quantile-python, ninja, mpmath, fastrlock, websockets, uvloop, urllib3, ujson, typing-extensions, tqdm, sympy, sniffio, shellingham, safetensors, rpds-py, regex, pyyaml, python-multipart, python-dotenv, pynvml, pygments, psutil, protobuf, packaging, orjson, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, msgpack, mdurl, MarkupSafe, idna, httptools, h11, fsspec, frozenlist, filelock, exceptiongroup, dnspython, click, charset-normalizer, certifi, attrs, annotated-types, uvicorn, triton, requests, referencing, pydantic-core, nvidia-cusparse-cu12, nvidia-cudnn-cu12, markdown-it-py, jinja2, httpcore, email_validator, cupy-cuda12x, anyio, aiosignal, aioprometheus, watchfiles, starlette, rich, pydantic, nvidia-cusolver-cu12, jsonschema-specifications, huggingface-hub, httpx, typer, torch, tokenizers, jsonschema, xformers, transformers, ray, fastapi-cli, fastapi, vllm
Successfully installed MarkupSafe-2.1.5 aioprometheus-23.12.0 aiosignal-1.3.1 annotated-types-0.7.0 anyio-4.4.0 attrs-23.2.0 certifi-2024.6.2 charset-normalizer-3.3.2 click-8.1.7 cupy-cuda12x-12.1.0 dnspython-2.6.1 email_validator-2.1.1 exceptiongroup-1.2.1 fastapi-0.111.0 fastapi-cli-0.0.4 fastrlock-0.8.2 filelock-3.14.0 frozenlist-1.4.1 fsspec-2024.6.0 h11-0.14.0 httpcore-1.0.5 httptools-0.6.1 httpx-0.27.0 huggingface-hub-0.23.3 idna-3.7 jinja2-3.1.4 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 markdown-it-py-3.0.0 mdurl-0.1.2 mpmath-1.3.0 msgpack-1.0.8 networkx-3.3 ninja-1.11.1.1 numpy-1.26.4 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.5.40 nvidia-nvtx-cu12-12.1.105 orjson-3.10.3 packaging-24.0 protobuf-5.27.0 psutil-5.9.8 pydantic-2.7.3 pydantic-core-2.18.4 pygments-2.18.0 pynvml-11.5.0 python-dotenv-1.0.1 python-multipart-0.0.9 pyyaml-6.0.1 quantile-python-1.1 ray-2.23.0 referencing-0.35.1 regex-2024.5.15 requests-2.32.3 rich-13.7.1 rpds-py-0.18.1 safetensors-0.4.3 sentencepiece-0.2.0 shellingham-1.5.4 sniffio-1.3.1 starlette-0.37.2 sympy-1.12.1 tokenizers-0.19.1 torch-2.1.2 tqdm-4.66.4 transformers-4.41.2 triton-2.1.0 typer-0.12.3 typing-extensions-4.12.1 ujson-5.10.0 urllib3-2.2.1 uvicorn-0.30.1 uvloop-0.19.0 vllm-0.3.2 watchfiles-0.22.0 websockets-12.0 xformers-0.0.23.post1
Collecting transformers==4.38.0
  Downloading transformers-4.38.0-py3-none-any.whl.metadata (131 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 131.1/131.1 kB 3.9 MB/s eta 0:00:00
Requirement already satisfied: filelock in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (3.14.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (0.23.3)
Requirement already satisfied: numpy>=1.17 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (24.0)
Requirement already satisfied: pyyaml>=5.1 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (6.0.1)
Requirement already satisfied: regex!=2019.12.17 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (2024.5.15)
Requirement already satisfied: requests in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (2.32.3)
Collecting tokenizers<0.19,>=0.14 (from transformers==4.38.0)
  Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
Requirement already satisfied: safetensors>=0.4.1 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (0.4.3)
Requirement already satisfied: tqdm>=4.27 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from transformers==4.38.0) (4.66.4)
Requirement already satisfied: fsspec>=2023.5.0 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.0) (2024.6.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.0) (4.12.1)
Requirement already satisfied: charset-normalizer<4,>=2 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from requests->transformers==4.38.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from requests->transformers==4.38.0) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from requests->transformers==4.38.0) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /home/azureuser/miniconda3/envs/qwen/lib/python3.10/site-packages (from requests->transformers==4.38.0) (2024.6.2)
Downloading transformers-4.38.0-py3-none-any.whl (8.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.5/8.5 MB 105.7 MB/s eta 0:00:00
Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 109.7 MB/s eta 0:00:00
Installing collected packages: tokenizers, transformers
  Attempting uninstall: tokenizers
    Found existing installation: tokenizers 0.19.1
    Uninstalling tokenizers-0.19.1:
      Successfully uninstalled tokenizers-0.19.1
  Attempting uninstall: transformers
    Found existing installation: transformers 4.41.2
    Uninstalling transformers-4.41.2:
      Successfully uninstalled transformers-4.41.2
Successfully installed tokenizers-0.15.2 transformers-4.38.0
[skypilot.yaml] Done installing packages.
I 06-05 10:55:55 cloud_vm_ray_backend.py:3228] Setup completed.
I 06-05 10:55:57 cloud_vm_ray_backend.py:3315] Job submitted with Job ID: 1
I 06-05 17:55:58 log_lib.py:408] Start streaming logs for job 1.
INFO: Tip: use Ctrl-C to exit log streaming (task will not be killed).
INFO: Waiting for task resources on 1 node. This will block if the cluster is full.
INFO: All task resources reserved.
INFO: Reserved IPs: ['<redacted>']
(task, pid=14418) [skypilot.yaml] Listing available conda environments:
(task, pid=14418) # conda environments:
(task, pid=14418) #
(task, pid=14418) base                  *  /home/azureuser/miniconda3
(task, pid=14418) qwen                     /home/azureuser/miniconda3/envs/qwen
(task, pid=14418) 
(task, pid=14418) [skypilot.yaml] Activating conda environment 'qwen'
(task, pid=14418) [skypilot.yaml] Listing available conda environments:
(task, pid=14418) # conda environments:
(task, pid=14418) #
(task, pid=14418) base                     /home/azureuser/miniconda3
(task, pid=14418) qwen                  *  /home/azureuser/miniconda3/envs/qwen
(task, pid=14418) 
(task, pid=14418) [skypilot.yaml] Listing installed packages:
(task, pid=14418) Package                                 Version
(task, pid=14418) --------------------------------------- -------------------
(task, pid=14418) adal                                    1.2.7
(task, pid=14418) aiohttp                                 3.9.5
(task, pid=14418) aiohttp-cors                            0.7.0
(task, pid=14418) aiosignal                               1.3.1
(task, pid=14418) annotated-types                         0.7.0
(task, pid=14418) antlr4-python3-runtime                  4.13.1
(task, pid=14418) applicationinsights                     0.11.10
(task, pid=14418) archspec                                0.2.1
(task, pid=14418) argcomplete                             3.3.0
(task, pid=14418) async-timeout                           4.0.3
(task, pid=14418) attrs                                   23.2.0
(task, pid=14418) azure-appconfiguration                  1.1.1
(task, pid=14418) azure-batch                             14.2.0
(task, pid=14418) azure-cli                               2.61.0
(task, pid=14418) azure-cli-core                          2.61.0
(task, pid=14418) azure-cli-telemetry                     1.1.0
(task, pid=14418) azure-common                            1.1.28
(task, pid=14418) azure-core                              1.30.1
(task, pid=14418) azure-cosmos                            3.2.0
(task, pid=14418) azure-data-tables                       12.4.0
(task, pid=14418) azure-datalake-store                    0.0.53
(task, pid=14418) azure-graphrbac                         0.60.0
(task, pid=14418) azure-identity                          1.16.0
(task, pid=14418) azure-keyvault-administration           4.4.0b2
(task, pid=14418) azure-keyvault-certificates             4.7.0
(task, pid=14418) azure-keyvault-keys                     4.9.0b3
(task, pid=14418) azure-keyvault-secrets                  4.7.0
(task, pid=14418) azure-mgmt-advisor                      9.0.0
(task, pid=14418) azure-mgmt-apimanagement                4.0.0
(task, pid=14418) azure-mgmt-appconfiguration             3.0.0
(task, pid=14418) azure-mgmt-appcontainers                2.0.0
(task, pid=14418) azure-mgmt-applicationinsights          1.0.0
(task, pid=14418) azure-mgmt-authorization                4.0.0
(task, pid=14418) azure-mgmt-batch                        17.3.0
(task, pid=14418) azure-mgmt-batchai                      7.0.0b1
(task, pid=14418) azure-mgmt-billing                      6.0.0
(task, pid=14418) azure-mgmt-botservice                   2.0.0
(task, pid=14418) azure-mgmt-cdn                          12.0.0
(task, pid=14418) azure-mgmt-cognitiveservices            13.5.0
(task, pid=14418) azure-mgmt-compute                      31.0.0
(task, pid=14418) azure-mgmt-containerinstance            10.1.0
(task, pid=14418) azure-mgmt-containerregistry            10.3.0
(task, pid=14418) azure-mgmt-containerservice             30.0.0
(task, pid=14418) azure-mgmt-core                         1.4.0
(task, pid=14418) azure-mgmt-cosmosdb                     9.4.0
(task, pid=14418) azure-mgmt-databoxedge                  1.0.0
(task, pid=14418) azure-mgmt-datalake-nspkg               3.0.1
(task, pid=14418) azure-mgmt-datalake-store               0.5.0
(task, pid=14418) azure-mgmt-datamigration                10.0.0
(task, pid=14418) azure-mgmt-devtestlabs                  4.0.0
(task, pid=14418) azure-mgmt-dns                          8.0.0
(task, pid=14418) azure-mgmt-eventgrid                    10.2.0b2
(task, pid=14418) azure-mgmt-eventhub                     10.1.0
(task, pid=14418) azure-mgmt-extendedlocation             1.0.0b2
(task, pid=14418) azure-mgmt-hdinsight                    9.0.0
(task, pid=14418) azure-mgmt-imagebuilder                 1.3.0
(task, pid=14418) azure-mgmt-iotcentral                   10.0.0b2
(task, pid=14418) azure-mgmt-iothub                       3.0.0
(task, pid=14418) azure-mgmt-iothubprovisioningservices   1.1.0
(task, pid=14418) azure-mgmt-keyvault                     10.3.0
(task, pid=14418) azure-mgmt-kusto                        0.3.0
(task, pid=14418) azure-mgmt-loganalytics                 13.0.0b4
(task, pid=14418) azure-mgmt-managedservices              1.0.0
(task, pid=14418) azure-mgmt-managementgroups             1.0.0
(task, pid=14418) azure-mgmt-maps                         2.0.0
(task, pid=14418) azure-mgmt-marketplaceordering          1.1.0
(task, pid=14418) azure-mgmt-media                        9.0.0
(task, pid=14418) azure-mgmt-monitor                      5.0.1
(task, pid=14418) azure-mgmt-msi                          7.0.0
(task, pid=14418) azure-mgmt-netapp                       10.1.0
(task, pid=14418) azure-mgmt-network                      25.4.0
(task, pid=14418) azure-mgmt-nspkg                        3.0.2
(task, pid=14418) azure-mgmt-policyinsights               1.1.0b4
(task, pid=14418) azure-mgmt-privatedns                   1.0.0
(task, pid=14418) azure-mgmt-rdbms                        10.2.0b17
(task, pid=14418) azure-mgmt-recoveryservices             3.0.0
(task, pid=14418) azure-mgmt-recoveryservicesbackup       9.1.0
(task, pid=14418) azure-mgmt-redhatopenshift              1.4.0
(task, pid=14418) azure-mgmt-redis                        14.3.0
(task, pid=14418) azure-mgmt-resource                     23.1.1
(task, pid=14418) azure-mgmt-search                       9.1.0
(task, pid=14418) azure-mgmt-security                     6.0.0
(task, pid=14418) azure-mgmt-servicebus                   8.2.0
(task, pid=14418) azure-mgmt-servicefabric                2.1.0
(task, pid=14418) azure-mgmt-servicefabricmanagedclusters 2.0.0b6
(task, pid=14418) azure-mgmt-servicelinker                1.2.0b2
(task, pid=14418) azure-mgmt-signalr                      2.0.0b1
(task, pid=14418) azure-mgmt-sql                          4.0.0b16
(task, pid=14418) azure-mgmt-sqlvirtualmachine            1.0.0b5
(task, pid=14418) azure-mgmt-storage                      21.1.0
(task, pid=14418) azure-mgmt-synapse                      2.1.0b5
(task, pid=14418) azure-mgmt-trafficmanager               1.0.0
(task, pid=14418) azure-mgmt-web                          7.2.0
(task, pid=14418) azure-monitor-query                     1.2.0
(task, pid=14418) azure-multiapi-storage                  1.2.0
(task, pid=14418) azure-nspkg                             3.0.2
(task, pid=14418) azure-storage-common                    1.4.2
(task, pid=14418) azure-synapse-accesscontrol             0.5.0
(task, pid=14418) azure-synapse-artifacts                 0.18.0
(task, pid=14418) azure-synapse-managedprivateendpoints   0.4.0
(task, pid=14418) azure-synapse-spark                     0.2.0
(task, pid=14418) bcrypt                                  4.1.3
(task, pid=14418) blessed                                 1.20.0
(task, pid=14418) boltons                                 23.0.0
(task, pid=14418) Brotli                                  1.0.9
(task, pid=14418) cachetools                              5.3.3
(task, pid=14418) certifi                                 2023.11.17
(task, pid=14418) cffi                                    1.16.0
(task, pid=14418) chardet                                 5.2.0
(task, pid=14418) charset-normalizer                      2.0.4
(task, pid=14418) click                                   8.1.7
(task, pid=14418) colorama                                0.4.6
(task, pid=14418) colorful                                0.5.6
(task, pid=14418) conda                                   23.11.0
(task, pid=14418) conda-content-trust                     0.2.0
(task, pid=14418) conda-libmamba-solver                   23.12.0
(task, pid=14418) conda-package-handling                  2.2.0
(task, pid=14418) conda_package_streaming                 0.9.0
(task, pid=14418) cryptography                            41.0.7
(task, pid=14418) decorator                               5.1.1
(task, pid=14418) Deprecated                              1.2.14
(task, pid=14418) distlib                                 0.3.8
(task, pid=14418) distro                                  1.8.0
(task, pid=14418) fabric                                  3.2.2
(task, pid=14418) filelock                                3.14.0
(task, pid=14418) frozenlist                              1.4.1
(task, pid=14418) google-api-core                         2.19.0
(task, pid=14418) google-auth                             2.29.0
(task, pid=14418) googleapis-common-protos                1.63.1
(task, pid=14418) gpustat                                 1.1.1
(task, pid=14418) grpcio                                  1.51.3
(task, pid=14418) humanfriendly                           10.0
(task, pid=14418) idna                                    3.4
(task, pid=14418) invoke                                  2.2.0
(task, pid=14418) isodate                                 0.6.1
(task, pid=14418) javaproperties                          0.5.2
(task, pid=14418) Jinja2                                  3.1.4
(task, pid=14418) jmespath                                1.0.1
(task, pid=14418) jsondiff                                2.0.0
(task, pid=14418) jsonpatch                               1.32
(task, pid=14418) jsonpointer                             2.1
(task, pid=14418) jsonschema                              4.22.0
(task, pid=14418) jsonschema-specifications               2023.12.1
(task, pid=14418) knack                                   0.11.0
(task, pid=14418) libmambapy                              1.5.3
(task, pid=14418) markdown-it-py                          3.0.0
(task, pid=14418) MarkupSafe                              2.1.5
(task, pid=14418) mdurl                                   0.1.2
(task, pid=14418) menuinst                                2.0.1
(task, pid=14418) msal                                    1.28.0
(task, pid=14418) msal-extensions                         1.2.0b1
(task, pid=14418) msgpack                                 1.0.8
(task, pid=14418) msrest                                  0.7.1
(task, pid=14418) msrestazure                             0.6.4.post1
(task, pid=14418) multidict                               6.0.5
(task, pid=14418) networkx                                3.3
(task, pid=14418) numpy                                   1.26.4
(task, pid=14418) nvidia-ml-py                            12.555.43
(task, pid=14418) oauthlib                                3.2.2
(task, pid=14418) opencensus                              0.11.4
(task, pid=14418) opencensus-context                      0.1.3
(task, pid=14418) packaging                               23.1
(task, pid=14418) pandas                                  2.2.2
(task, pid=14418) paramiko                                3.4.0
(task, pid=14418) pendulum                                3.0.0
(task, pid=14418) pip                                     23.0.1
(task, pid=14418) pkginfo                                 1.11.0
(task, pid=14418) platformdirs                            3.10.0
(task, pid=14418) pluggy                                  1.0.0
(task, pid=14418) portalocker                             2.8.2
(task, pid=14418) prettytable                             3.10.0
(task, pid=14418) prometheus_client                       0.20.0
(task, pid=14418) proto-plus                              1.23.0
(task, pid=14418) protobuf                                4.25.3
(task, pid=14418) psutil                                  5.9.8
(task, pid=14418) PuLP                                    2.8.0
(task, pid=14418) py-spy                                  0.3.14
(task, pid=14418) pyasn1                                  0.6.0
(task, pid=14418) pyasn1_modules                          0.4.0
(task, pid=14418) pycomposefile                           0.0.31
(task, pid=14418) pycosat                                 0.6.6
(task, pid=14418) pycparser                               2.21
(task, pid=14418) pydantic                                2.7.3
(task, pid=14418) pydantic_core                           2.18.4
(task, pid=14418) PyGithub                                1.59.1
(task, pid=14418) Pygments                                2.18.0
(task, pid=14418) PyJWT                                   2.8.0
(task, pid=14418) PyNaCl                                  1.5.0
(task, pid=14418) pyOpenSSL                               23.2.0
(task, pid=14418) PySocks                                 1.7.1
(task, pid=14418) python-dateutil                         2.9.0.post0
(task, pid=14418) python-dotenv                           1.0.1
(task, pid=14418) pytz                                    2024.1
(task, pid=14418) PyYAML                                  6.0.1
(task, pid=14418) ray                                     2.9.3
(task, pid=14418) referencing                             0.35.1
(task, pid=14418) requests                                2.31.0
(task, pid=14418) requests-oauthlib                       2.0.0
(task, pid=14418) rich                                    13.7.1
(task, pid=14418) rpds-py                                 0.18.1
(task, pid=14418) rsa                                     4.9
(task, pid=14418) ruamel.yaml                             0.17.21
(task, pid=14418) ruamel.yaml.clib                        0.2.6
(task, pid=14418) scp                                     0.13.6
(task, pid=14418) semver                                  2.13.0
(task, pid=14418) setuptools                              65.5.0
(task, pid=14418) six                                     1.16.0
(task, pid=14418) skypilot                                1.0.0.dev2024053101
(task, pid=14418) smart-open                              7.0.4
(task, pid=14418) sshtunnel                               0.1.5
(task, pid=14418) tabulate                                0.9.0
(task, pid=14418) time-machine                            2.14.1
(task, pid=14418) tqdm                                    4.65.0
(task, pid=14418) truststore                              0.8.0
(task, pid=14418) typing_extensions                       4.12.1
(task, pid=14418) tzdata                                  2024.1
(task, pid=14418) urllib3                                 1.26.18
(task, pid=14418) virtualenv                              20.26.2
(task, pid=14418) wcwidth                                 0.2.13
(task, pid=14418) websocket-client                        1.3.3
(task, pid=14418) wheel                                   0.41.2
(task, pid=14418) wrapt                                   1.16.0
(task, pid=14418) xmltodict                               0.13.0
(task, pid=14418) yarl                                    1.9.4
(task, pid=14418) zstandard                               0.19.0
(task, pid=14418) 
(task, pid=14418) [notice] A new release of pip is available: 23.0.1 -> 24.0
(task, pid=14418) [notice] To update, run: pip install --upgrade pip
(task, pid=14418) [skypilot.yaml] Setting PATH to include /sbin
(task, pid=14418) [skypilot.yaml] Starting vllm OpenAI API server with the following configuration:
(task, pid=14418) [skypilot.yaml]   - Host: 0.0.0.0
(task, pid=14418) [skypilot.yaml]   - Model: Qwen/Qwen1.5-7B-Chat
(task, pid=14418) [skypilot.yaml]   - Tensor Parallel Size: 1
(task, pid=14418) [skypilot.yaml]   - Maximum Model Length: 1024
(task, pid=14418) /home/azureuser/skypilot-runtime/bin/python: Error while finding module specification for 'vllm.entrypoints.openai.api_server' (ModuleNotFoundError: No module named 'vllm')
INFO: Job finished (status: SUCCEEDED).
I 06-05 10:56:03 cloud_vm_ray_backend.py:3350] Job ID: 1
I 06-05 10:56:03 cloud_vm_ray_backend.py:3350] To cancel the job:       sky cancel qwen 1
I 06-05 10:56:03 cloud_vm_ray_backend.py:3350] To stream job logs:      sky logs qwen 1
I 06-05 10:56:03 cloud_vm_ray_backend.py:3350] To view the job queue:   sky queue qwen
I 06-05 10:56:03 cloud_vm_ray_backend.py:3446] 
I 06-05 10:56:03 cloud_vm_ray_backend.py:3446] Cluster name: qwen
I 06-05 10:56:03 cloud_vm_ray_backend.py:3446] To log into the head VM: ssh qwen
I 06-05 10:56:03 cloud_vm_ray_backend.py:3446] To submit a job:         sky exec qwen yaml_file
I 06-05 10:56:03 cloud_vm_ray_backend.py:3446] To stop the cluster:     sky stop qwen
I 06-05 10:56:03 cloud_vm_ray_backend.py:3446] To teardown the cluster: sky down qwen
Clusters
NAME  LAUNCHED    RESOURCES                                                                  STATUS  AUTOSTOP  COMMAND                       
qwen  3 mins ago  1x Azure(Standard_NC24ads_A100_v4, {'A100-80GB': 1}, disk_tier=best, p...  UP      -         sky launch -c qwen x.yaml...

[Michaelvll]

Thanks for filing the issue @WesleyYue! This was indeed a bug, and should be fixed by #3639 in our master branch now. Could you try to run it again with the latest master?

[WesleyYue]

Yep it's fixed. Thanks for the quick fix! This is probably one of the highest quality OSS projects I've used. Documentation is excellent!

[richard-alexander]

I'm having this exact same error. When I SSH into the remote server, I see all modules are installed. But when I sky exec, I get a module not found error on the very first import.

Has this fix been released? Pushed to PyPi? Or was it reintroduced somehow?

[Michaelvll]

I'm having this exact same error. When I SSH into the remote server, I see all modules are installed. But when I sky exec, I get a module not found error on the very first import.

Has this fix been released? Pushed to PyPi? Or was it reintroduced somehow?

Hi @richard-alexander, what version of SkyPilot are you currently working on? This information can be get with sky -v and sky -c. You could try to use the latest nightly build to get the fix pip uninstall skypilot; pip install -U skypilot-nightly.

[richard-alexander]

I'm on:

skypilot, version 0.6.0
skypilot, commit e37a39d19a8f5b8fd7809e29c8691f65e462896a

commit e37a39d

I'll try nightly now...

[richard-alexander]

Looks like nightly fixed it.
Thanks a ton for the quick response @Michaelvll! I really appreciate the work you're doing with sky-pilot! :-D