[Installation]: Installing vllm in GH200 machine (aarch64) causes problems with cusparse.h missing

[tajwarfahim]

Your current environment

I cannot run collect_env.py since that would require vllm

How you are installing vllm

I am following the instructions from here:
https://docs.vllm.ai/en/stable/getting_started/installation.html#use-an-existing-pytorch-installation

Problem I am facing:

pip install . --verbose --no-build-isolation
Using pip 24.3.1 from /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/pip (python 3.10)
Processing /work/nvme/bcfp/ftajwar/vllm
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info
  writing /tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/dependency_links.txt
  writing entry points to /tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/entry_points.txt
  writing requirements to /tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/requires.txt
  writing top-level names to /tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'LICENSE'
  writing manifest file '/tmp/pip-modern-metadata-nus2ddwg/vllm.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-nus2ddwg/vllm-0.6.4.post2.dev359+g4863e5fb.d20241214.cu123.dist-info'
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: psutil in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (6.1.0)
Requirement already satisfied: sentencepiece in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.2.0)
Requirement already satisfied: numpy<2.0.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.26.3)
Requirement already satisfied: requests>=2.26.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.32.3)
Requirement already satisfied: tqdm in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (4.67.1)
Collecting blake3 (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123)
  Obtaining dependency information for blake3 from https://files.pythonhosted.org/packages/1a/9c/bf926c668066ab3875d2ac588d1c29424324303aad9138f6476370753348/blake3-1.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata
  Using cached blake3-1.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (4.1 kB)
Requirement already satisfied: py-cpuinfo in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (9.0.0)
Requirement already satisfied: transformers>=4.45.2 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (4.46.3)
Requirement already satisfied: tokenizers>=0.19.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.20.3)
Requirement already satisfied: protobuf in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (5.29.1)
Requirement already satisfied: fastapi!=0.113.*,!=0.114.0,>=0.107.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.115.6)
Requirement already satisfied: aiohttp in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.11.10)
Requirement already satisfied: openai>=1.45.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.57.2)
Requirement already satisfied: uvicorn[standard] in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.32.1)
Requirement already satisfied: pydantic>=2.9 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.10.3)
Requirement already satisfied: pillow in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (10.4.0)
Requirement already satisfied: prometheus_client>=0.18.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.21.1)
Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (7.0.0)
Requirement already satisfied: tiktoken>=0.6.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.7.0)
Requirement already satisfied: lm-format-enforcer<0.11,>=0.10.9 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.10.9)
Collecting outlines==0.1.9 (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123)
  Obtaining dependency information for outlines==0.1.9 from https://files.pythonhosted.org/packages/9f/24/ffa97943aee9e7f1cdad76f57ab7679df283c593be0fbdd5eda711a4d705/outlines-0.1.9-py3-none-any.whl.metadata
  Using cached outlines-0.1.9-py3-none-any.whl.metadata (17 kB)
Requirement already satisfied: typing_extensions>=4.10 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (4.12.2)
Requirement already satisfied: filelock>=3.16.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.16.1)
Requirement already satisfied: partial-json-parser in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.2.1.1.post4)
Requirement already satisfied: pyzmq in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (26.2.0)
Requirement already satisfied: msgspec in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.18.6)
Requirement already satisfied: gguf==0.10.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.10.0)
Requirement already satisfied: importlib_metadata in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (8.5.0)
Requirement already satisfied: mistral_common>=1.5.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.5.1)
Requirement already satisfied: pyyaml in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (6.0.2)
Requirement already satisfied: einops in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.8.0)
Requirement already satisfied: compressed-tensors==0.8.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.8.0)
Requirement already satisfied: ray>=2.9 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.40.0)
Requirement already satisfied: nvidia-ml-py>=12.560.30 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (12.560.30)
Requirement already satisfied: torch>=1.7.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from compressed-tensors==0.8.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.6.0.dev20241210+cu124)
Requirement already satisfied: interegular in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.3.3)
Requirement already satisfied: jinja2 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.1.3)
Requirement already satisfied: lark in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.2.2)
Requirement already satisfied: nest_asyncio in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.6.0)
Requirement already satisfied: cloudpickle in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.1.0)
Requirement already satisfied: diskcache in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (5.6.3)
Requirement already satisfied: referencing in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.35.1)
Requirement already satisfied: jsonschema in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (4.23.0)
Requirement already satisfied: pycountry in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (24.6.1)
Requirement already satisfied: airportsdata in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (20241001)
Collecting outlines_core==0.1.24 (from outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123)
  Using cached outlines_core-0.1.24-cp310-cp310-linux_aarch64.whl
Requirement already satisfied: starlette<0.42.0,>=0.40.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from fastapi!=0.113.*,!=0.114.0,>=0.107.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.41.3)
Requirement already satisfied: packaging in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from lm-format-enforcer<0.11,>=0.10.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (24.2)
Requirement already satisfied: opencv-python-headless<5.0.0,>=4.0.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from mistral_common[opencv]>=1.5.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (4.10.0.84)
Requirement already satisfied: anyio<5,>=3.5.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (4.7.0)
Requirement already satisfied: distro<2,>=1.7.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.28.1)
Requirement already satisfied: jiter<1,>=0.4.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.8.2)
Requirement already satisfied: sniffio in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.3.1)
Requirement already satisfied: annotated-types>=0.6.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from pydantic>=2.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.7.0)
Requirement already satisfied: pydantic-core==2.27.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from pydantic>=2.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.27.1)
Requirement already satisfied: click>=7.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from ray>=2.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (8.1.7)
Requirement already satisfied: msgpack<2.0.0,>=1.0.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from ray>=2.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.1.0)
Requirement already satisfied: aiosignal in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from ray>=2.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.3.1)
Requirement already satisfied: frozenlist in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from ray>=2.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.5.0)
Requirement already satisfied: charset-normalizer<4,>=2 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.26.13)
Requirement already satisfied: certifi>=2017.4.17 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from requests>=2.26.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2022.12.7)
Requirement already satisfied: regex>=2022.1.18 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from tiktoken>=0.6.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2024.11.6)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from tokenizers>=0.19.1->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.26.5)
Requirement already satisfied: safetensors>=0.4.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from transformers>=4.45.2->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.4.5)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from aiohttp->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.4.4)
Requirement already satisfied: async-timeout<6.0,>=4.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from aiohttp->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (5.0.1)
Requirement already satisfied: attrs>=17.3.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from aiohttp->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (24.2.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from aiohttp->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (6.1.0)
Requirement already satisfied: propcache>=0.2.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from aiohttp->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.2.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from aiohttp->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.18.3)
Requirement already satisfied: zipp>=3.20 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from importlib_metadata->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.21.0)
Requirement already satisfied: h11>=0.8 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.14.0)
Requirement already satisfied: httptools>=0.6.3 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.6.4)
Requirement already satisfied: python-dotenv>=0.13 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.0.1)
Requirement already satisfied: uvloop!=0.15.0,!=0.15.1,>=0.14.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.21.0)
Requirement already satisfied: watchfiles>=0.13 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.0.3)
Requirement already satisfied: websockets>=10.4 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from uvicorn[standard]->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (14.1)
Requirement already satisfied: exceptiongroup>=1.0.2 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from anyio<5,>=3.5.0->openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.2.2)
Requirement already satisfied: httpcore==1.* in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from httpx<1,>=0.23.0->openai>=1.45.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.0.7)
Requirement already satisfied: fsspec>=2023.5.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.19.1->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2024.9.0)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from jsonschema->outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2024.10.1)
Requirement already satisfied: rpds-py>=0.7.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from jsonschema->outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (0.22.3)
Requirement already satisfied: sympy==1.13.1 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from torch>=1.7.0->compressed-tensors==0.8.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.13.1)
Requirement already satisfied: networkx in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from torch>=1.7.0->compressed-tensors==0.8.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (3.2.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.7.0->compressed-tensors==0.8.0->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages (from jinja2->outlines==0.1.9->vllm==0.6.4.post2.dev359+g4863e5fb.d20241214.cu123) (2.1.5)
Using cached outlines-0.1.9-py3-none-any.whl (87 kB)
Using cached blake3-1.0.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (354 kB)
Building wheels for collected packages: vllm
  Running command Building wheel for vllm (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  copying vllm/_version.py -> build/lib.linux-aarch64-cpython-310/vllm
  running build_ext
  Using MAX_JOBS=1 as the number of jobs.
  -- Build type: RelWithDebInfo
  -- Target device: cuda
  -- Found python matching: /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/bin/python3.10.
  -- PyTorch: CUDA detected: 12.3
  -- PyTorch: CUDA nvcc is: /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/bin/nvcc
  -- PyTorch: CUDA toolkit directory: /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3
  -- PyTorch: Header version is: 12.3
  -- Found Python: /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/bin/python3.10 (found version "3.10.16") found components: Interpreter
  CMake Warning at /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:140 (message):
    Failed to compute shorthash for libnvrtc.so
  Call Stack (most recent call first):
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
    CMakeLists.txt:84 (find_package)


  CMake Warning (dev) at /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/cmake/data/share/cmake-3.31/Modules/FindPackageHandleStandardArgs.cmake:441 (message):
    The package name passed to `find_package_handle_standard_args` (nvtx3) does
    not match the name of the calling package (Caffe2).  This can lead to
    problems in calling code that expects `find_package` result variables
    (e.g., `_FOUND`) to follow a certain pattern.
  Call Stack (most recent call first):
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:178 (find_package_handle_standard_args)
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
    CMakeLists.txt:84 (find_package)
  This warning is for project developers.  Use -Wno-dev to suppress it.

  -- Could NOT find nvtx3 (missing: nvtx3_dir)
  CMake Warning at /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:184 (message):
    Cannot find NVTX3, find old NVTX instead
  Call Stack (most recent call first):
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:86 (include)
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
    CMakeLists.txt:84 (find_package)


  -- USE_CUDNN is set to 0. Compiling without cuDNN support
  -- USE_CUSPARSELT is set to 0. Compiling without cuSPARSELt support
  -- USE_CUDSS is set to 0. Compiling without cuDSS support
  -- USE_CUFILE is set to 0. Compiling without cuFile support
  -- Autodetected CUDA architecture(s):  9.0
  -- Added CUDA NVCC flags for: -gencode;arch=compute_90,code=sm_90
  CMake Warning at /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:22 (message):
    static library kineto_LIBRARY-NOTFOUND not found.
  Call Stack (most recent call first):
    /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/share/cmake/Torch/TorchConfig.cmake:121 (append_torchlib_if_found)
    CMakeLists.txt:84 (find_package)


  CMake Warning at CMakeLists.txt:107 (message):
    Pytorch version 2.5.1 expected for CUDA build, saw 2.6.0 instead.


  -- CUDA target architectures: 9.0
  -- CUDA supported target architectures: 9.0
  -- FetchContent base directory: /work/nvme/bcfp/ftajwar/vllm/.deps
  -- CMake Version: 3.31.1
  -- CUTLASS 3.5.1
  -- CUDART: /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/lib64/libcudart.so
  -- CUDA Driver: /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/lib64/stubs/libcuda.so
  -- NVRTC: /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/lib64/libnvrtc.so
  -- Default Install Location: install
  -- Make cute::tuple be the new standard-layout tuple type
  -- CUDA Compilation Architectures: 70;72;75;80;86;87;89;90;90a
  -- Enable caching of reference results in conv unit tests
  -- Enable rigorous conv problem sizes in conv unit tests
  -- Using NVCC flags: --expt-relaxed-constexpr;-DCUTE_USE_PACKED_TUPLE=1;-DCUTLASS_TEST_LEVEL=0;-DCUTLASS_TEST_ENABLE_CACHED_RESULTS=1;-DCUTLASS_CONV_UNIT_TEST_RIGOROUS_SIZE_ENABLED=1;-DCUTLASS_DEBUG_TRACE_LEVEL=0;-Xcompiler=-Wconversion;-Xcompiler=-fno-strict-aliasing;-lineinfo
  -- Configuring cublas ...
  -- cuBLAS Disabled.
  -- Configuring cuBLAS ... done.
  -- Building Marlin kernels for archs: 9.0
  -- Building scaled_mm_c3x for archs: 9.0a;9.0
  -- Not building scaled_mm_c2x as all archs are already built for and covered by scaled_mm_c3x
  -- Machete generation script hash: c5413e168eabdf729344fe0f89eb8aa2
  -- Last run machete generate script hash: c5413e168eabdf729344fe0f89eb8aa2
  -- Machete generation script has not changed, skipping generation.
  -- Building Machete kernels for archs: 9.0a
  -- Enabling C extension.
  -- Building Marlin MOE kernels for archs: 9.0
  -- Enabling moe extension.
  -- Build type: RelWithDebInfo
  -- Target device: cuda
  -- Found Python: /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/bin/python3.10 (found version "3.10.16") found components: Interpreter Development.Module Development.SABIModule
  -- Building vllm-flash-attn inside vLLM. Skipping flag detection and relying on parent build.
  -- vllm-flash-attn is available at /work/nvme/bcfp/ftajwar/vllm/.deps/vllm-flash-attn-src
  -- Configuring done (6.3s)
  -- Generating done (0.1s)
  -- Build files have been written to: /work/nvme/bcfp/ftajwar/vllm/build/temp.linux-aarch64-cpython-310
  Using MAX_JOBS=1 as the number of jobs.
  [1/115] Building CUDA object CMakeFiles/_moe_C.dir/csrc/moe/moe_align_sum_kernels.cu.o
  FAILED: CMakeFiles/_moe_C.dir/csrc/moe/moe_align_sum_kernels.cu.o
  /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/bin/nvcc -forward-unknown-to-host-compiler -DPy_LIMITED_API=3 -DTORCH_EXTENSION_NAME=_moe_C -DUSE_C10D_GLOO -DUSE_C10D_NCCL -DUSE_DISTRIBUTED -DUSE_RPC -DUSE_TENSORPIPE -D_moe_C_EXPORTS -I/work/nvme/bcfp/ftajwar/vllm/csrc -isystem /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/include/python3.10 -isystem /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/include -isystem /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /opt/nvidia/hpc_sdk/Linux_aarch64/24.3/cuda/12.3/include -DONNX_NAMESPACE=onnx_c2 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -O2 -g -DNDEBUG -std=c++17 -Xcompiler=-fPIC --expt-relaxed-constexpr -DENABLE_FP8 --threads=1 -DENABLE_SCALED_MM_C3X=1 -D_GLIBCXX_USE_CXX11_ABI=1 -gencode arch=compute_90,code=sm_90 -MD -MT CMakeFiles/_moe_C.dir/csrc/moe/moe_align_sum_kernels.cu.o -MF CMakeFiles/_moe_C.dir/csrc/moe/moe_align_sum_kernels.cu.o.d -x cu -c /work/nvme/bcfp/ftajwar/vllm/csrc/moe/moe_align_sum_kernels.cu -o CMakeFiles/_moe_C.dir/csrc/moe/moe_align_sum_kernels.cu.o
  In file included from /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/include/ATen/cuda/CUDAContext.h:3,
                   from /work/nvme/bcfp/ftajwar/vllm/csrc/moe/moe_align_sum_kernels.cu:2:
  /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/torch/include/ATen/cuda/CUDAContextLight.h:7:10: fatal error: cusparse.h: No such file or directory
      7 | #include <cusparse.h>
        |          ^~~~~~~~~~~~
  compilation terminated.
  ninja: build stopped: subcommand failed.
  Traceback (most recent call last):
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/build_meta.py", line 438, in build_wheel
      return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/build_meta.py", line 426, in _build
      return self._build_with_temp_dir(
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
      self.run_setup()
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in run_setup
      exec(code, locals())
    File "<string>", line 600, in <module>
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 183, in setup
      return run_commands(dist)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
      dist.run_commands()
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
      self.run_command(cmd)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/dist.py", line 995, in run_command
      super().run_command(command)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/command/bdist_wheel.py", line 381, in run
      self.run_command("build")
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/dist.py", line 995, in run_command
      super().run_command(command)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/dist.py", line 995, in run_command
      super().run_command(command)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "<string>", line 238, in run
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 99, in run
      _build_ext.run(self)
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
      self.build_extensions()
    File "<string>", line 212, in build_extensions
    File "/work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['cmake', '--build', '.', '-j=1', '--target=_moe_C', '--target=vllm_flash_attn_c', '--target=_C']' returned non-zero exit status 1.
  error: subprocess-exited-with-error
  
  × Building wheel for vllm (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/bin/python3.10 /work/nvme/bcfp/ftajwar/anaconda3/envs/exploration/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmph8nwqrko
  cwd: /work/nvme/bcfp/ftajwar/vllm
  Building wheel for vllm (pyproject.toml) ... error
  ERROR: Failed building wheel for vllm
Failed to build vllm
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (vllm)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[JordanNanos]

consider @drikster80's image, built from nvidia-pytorch: #10499

[NickLucche]

Have you double checked your env vars to point to the right cuda installation (something like)

CUDA_HOME=/usr/local/cuda PATH="${CUDA_HOME}/bin:$PATH" pip install ...