as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2

[KLL535]

Help me!

(venv) user@DESKTOP-ULK7M66:~/Open-Sora$ pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git
Using pip 24.3.1 from /home/user/Open-Sora/venv/lib/python3.11/site-packages/pip (python 3.11)
Collecting git+https://github.com/NVIDIA/apex.git
Cloning https://github.com/NVIDIA/apex.git to /tmp/pip-req-build-0i9ipbc1
Running command git version
git version 2.43.0
Running command git clone --filter=blob:none https://github.com/NVIDIA/apex.git /tmp/pip-req-build-0i9ipbc1
Cloning into '/tmp/pip-req-build-0i9ipbc1'...
Updating files: 0% (2/503)

...

Updating files: 100% (503/503), done.
Running command git rev-parse HEAD
73375b3
Resolved https://github.com/NVIDIA/apex.git to commit 73375b3
Running command git submodule update --init --recursive -q
Running command git rev-parse HEAD
73375b3
Running command Preparing metadata (pyproject.toml)

torch.version = 2.2.2+cu121

running dist_info
creating /tmp/pip-modern-metadata-r_frcyiz/apex.egg-info
writing /tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/dependency_links.txt
writing requirements to /tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/requires.txt
writing top-level names to /tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/top_level.txt
writing manifest file '/tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file '/tmp/pip-modern-metadata-r_frcyiz/apex.egg-info/SOURCES.txt'
creating '/tmp/pip-modern-metadata-r_frcyiz/apex-0.1.dist-info'
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: packaging>20.6 in ./venv/lib/python3.11/site-packages (from apex==0.1) (24.2)
Building wheels for collected packages: apex
Running command Building wheel for apex (pyproject.toml)

torch.version = 2.2.2+cu121

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0
from /home/user/anaconda3/bin

running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-311/apex
copying apex/_autocast_utils.py -> build/lib.linux-x86_64-cpython-311/apex
copying apex/init.py -> build/lib.linux-x86_64-cpython-311/apex
creating build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/fused_adagrad.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/fused_sgd.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/init.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/fused_novograd.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/fused_lamb.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/fused_adam.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
copying apex/optimizers/fused_mixed_precision_lamb.py -> build/lib.linux-x86_64-cpython-311/apex/optimizers
creating build/lib.linux-x86_64-cpython-311/apex/amp
copying apex/amp/compat.py -> build/lib.linux-x86_64-cpython-311/apex/amp

...

copying apex/transformer/pipeline_parallel/schedules/fwd_bwd_no_pipelining.py -> build/lib.linux-x86_64-cpython-311/apex/transformer/pipeline_parallel/schedules
copying apex/transformer/pipeline_parallel/schedules/init.py -> build/lib.linux-x86_64-cpython-311/apex/transformer/pipeline_parallel/schedules
running build_ext
/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py:425: UserWarning: There are no x86_64-linux-gnu-g++ version bounds defined for CUDA version 12.1
warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
building 'apex_C' extension
creating /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc
Emitting ninja build file /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/1] c++ -MMD -MF /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/flatten_unflatten.o.d -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/flatten_unflatten.cpp -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/flatten_unflatten.o -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=apex_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
x86_64-linux-gnu-g++ -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -shared -Wl,-O1 -Wl,-Bsymbolic-functions /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/flatten_unflatten.o -L/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/lib -L/usr/lib/x86_64-linux-gnu -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-cpython-311/apex_C.cpython-311-x86_64-linux-gnu.so
building 'amp_C' extension
Emitting ninja build file /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/update_scale_hysteresis.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/update_scale_hysteresis.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/update_scale_hysteresis.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/update_scale_hysteresis.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/update_scale_hysteresis.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/update_scale_hysteresis.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/update_scale_hysteresis.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[2/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_2.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb_stage_2.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_2.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_2.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_2.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb_stage_2.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_2.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[3/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel_mp.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_l2norm_kernel_mp.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel_mp.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel_mp.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel_mp.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_l2norm_kernel_mp.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel_mp.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[4/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_novograd.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_novograd.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_novograd.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_novograd.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_novograd.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_novograd.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_novograd.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[5/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_scale_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_l2norm_scale_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_scale_kernel.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_scale_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_l2norm_scale_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[6/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adagrad.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_adagrad.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adagrad.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adagrad.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adagrad.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_adagrad.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adagrad.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[7/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_sgd_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_sgd_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_sgd_kernel.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_sgd_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_sgd_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_sgd_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[8/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_axpby_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_axpby_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_axpby_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_axpby_kernel.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_axpby_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_axpby_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_axpby_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[9/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[10/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_l2norm_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_l2norm_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_l2norm_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[11/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_scale_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_scale_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_scale_kernel.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_scale_kernel.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_scale_kernel.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_scale_kernel.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[12/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_1.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb_stage_1.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_1.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_1.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_1.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb_stage_1.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_stage_1.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[13/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_mp.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb_mp.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_mp.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_mp.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_mp.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_lamb_mp.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_lamb_mp.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[14/15] /home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adam.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_adam.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adam.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
FAILED: /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adam.o
/home/user/anaconda3/bin/nvcc --generate-dependencies-with-compile --dependency-output /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adam.o.d -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/multi_tensor_adam.cu -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/multi_tensor_adam.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -lineinfo -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17
as: symbol lookup error: as: undefined symbol: ZSTD_compressStream2
[15/15] c++ -MMD -MF /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/amp_C_frontend.o.d -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/TH -I/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/include/THC -I/home/user/anaconda3/include -I/home/user/Open-Sora/venv/include -I/usr/include/python3.11 -c -c /tmp/pip-req-build-0i9ipbc1/csrc/amp_C_frontend.cpp -o /tmp/pip-req-build-0i9ipbc1/build/temp.linux-x86_64-cpython-311/csrc/amp_C_frontend.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=amp_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++17
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2096, in _run_ninja_build
subprocess.run(
File "/usr/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in
main()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 438, in build_wheel
return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build
return self._build_with_temp_dir(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
self.run_setup()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
exec(code, locals())
File "", line 862, in
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/init.py", line 117, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 183, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
dist.run_commands()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
self.run_command(cmd)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
super().run_command(command)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
cmd_obj.run()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 381, in run
self.run_command("build")
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
super().run_command(command)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
cmd_obj.run()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
self.run_command(cmd_name)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
self.distribution.run_command(command)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
super().run_command(command)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
cmd_obj.run()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 99, in run
_build_ext.run(self)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
self.build_extensions()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 871, in build_extensions
build_ext.build_extensions(self)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
self._build_extensions_serial()
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
self.build_extension(ext)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 264, in build_extension
_build_ext.build_extension(self, ext)
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 557, in build_extension
objects = self.compiler.compile(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 684, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1774, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/home/user/Open-Sora/venv/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 2112, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error compiling objects for extension
error: subprocess-exited-with-error

× Building wheel for apex (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/user/Open-Sora/venv/bin/python /home/user/Open-Sora/venv/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpbt5dhtu2
cwd: /tmp/pip-req-build-0i9ipbc1
Building wheel for apex (pyproject.toml) ... error
ERROR: Failed building wheel for apex
Failed to build apex
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (apex)