Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure with EMU3 T2I Inference. #309

Open
lambert-x opened this issue Jan 6, 2025 · 8 comments
Open

Failure with EMU3 T2I Inference. #309

lambert-x opened this issue Jan 6, 2025 · 8 comments

Comments

@lambert-x
Copy link

Hi, thanks for the implementation of EMU3. I tried to run the generation script but meet the error message showing below.

codes/FlagScale/flagscale/inference/core/scheduler.py", line 1020, in _schedule_prefills
[rank0]:     assert num_new_tokens == num_prompt_tokens
[rank0]: AssertionError

**Conda environment info: **

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
absl-py 2.1.0 pypi_0 pypi
accelerate 1.2.1 pypi_0 pypi
aiofiles 23.2.1 pypi_0 pypi
aiohappyeyeballs 2.4.4 py310h06a4308_0
aiohttp 3.11.10 py310h5eee18b_0
aiosignal 1.2.0 pyhd3eb1b0_0
aniso8601 9.0.1 pypi_0 pypi
annotated-types 0.7.0 pypi_0 pypi
antlr4-python3-runtime 4.9.3 pypi_0 pypi
anyio 4.8.0 pypi_0 pypi
argcomplete 3.5.3 pypi_0 pypi
argon2-cffi 23.1.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
arrow 1.3.0 pypi_0 pypi
arrow-cpp 17.0.0 h865e1df_1
asciitree 0.3.3 pypi_0 pypi
astroid 3.2.4 pypi_0 pypi
asttokens 3.0.0 pypi_0 pypi
async-lru 2.0.4 pypi_0 pypi
async-timeout 5.0.1 py310h06a4308_0
attrs 24.3.0 py310h06a4308_0
audioread 3.0.1 pypi_0 pypi
aws-c-auth 0.6.19 h5eee18b_0
aws-c-cal 0.5.20 hdbd6064_0
aws-c-common 0.8.5 h5eee18b_0
aws-c-compression 0.2.16 h5eee18b_0
aws-c-event-stream 0.2.15 h6a678d5_0
aws-c-http 0.6.25 h5eee18b_0
aws-c-io 0.13.10 h5eee18b_0
aws-c-mqtt 0.7.13 h5eee18b_0
aws-c-s3 0.1.51 hdbd6064_0
aws-c-sdkutils 0.1.6 h5eee18b_0
aws-checksums 0.1.13 h5eee18b_0
aws-crt-cpp 0.18.16 h6a678d5_0
aws-sdk-cpp 1.11.212 hecad206_0
awscli 1.36.33 pypi_0 pypi
babel 2.16.0 pypi_0 pypi
beautifulsoup4 4.12.3 pypi_0 pypi
bitsandbytes 0.45.0 pypi_0 pypi
black 24.4.2 pypi_0 pypi
blas 1.0 mkl
bleach 6.2.0 pypi_0 pypi
blinker 1.9.0 pypi_0 pypi
boost-cpp 1.82.0 hdb19cb5_2
boto3 1.35.92 pypi_0 pypi
botocore 1.35.92 pypi_0 pypi
bottleneck 1.4.2 py310ha9d4c09_0
braceexpand 0.1.7 pypi_0 pypi
brotli-python 1.0.9 py310h6a678d5_8
buildkite-test-collector 0.1.8 pypi_0 pypi
bzip2 1.0.8 h5eee18b_6
c-ares 1.19.1 h5eee18b_0
ca-certificates 2024.11.26 h06a4308_0
certifi 2024.12.14 py310h06a4308_0
cffi 1.17.1 pypi_0 pypi
cfgv 3.4.0 pypi_0 pypi
chardet 5.2.0 pypi_0 pypi
charset-normalizer 3.4.1 pypi_0 pypi
clang-format 18.1.5 pypi_0 pypi
click 8.1.8 pypi_0 pypi
cloudpickle 3.1.0 pypi_0 pypi
cmake 3.31.2 pypi_0 pypi
codespell 2.3.0 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
comm 0.2.2 pypi_0 pypi
compressed-tensors 0.8.0 pypi_0 pypi
contourpy 1.3.1 pypi_0 pypi
coverage 7.6.10 pypi_0 pypi
crcmod 1.7 pypi_0 pypi
cuda-cudart 12.1.105 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-libraries 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-opencl 12.6.77 0 nvidia
cuda-runtime 12.1.0 0 nvidia
cuda-version 12.6 3 nvidia
cupy-cuda12x 13.3.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
datamodel-code-generator 0.26.4 pypi_0 pypi
dataproperty 1.1.0 pypi_0 pypi
datasets 2.19.1 py310h06a4308_0
debugpy 1.8.11 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
deepspeed 0.16.2 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
diffusers 0.32.1 pypi_0 pypi
dill 0.3.8 py310h06a4308_0
diskcache 5.6.3 pypi_0 pypi
distlib 0.3.9 pypi_0 pypi
distro 1.9.0 pypi_0 pypi
dnspython 2.7.0 pypi_0 pypi
docker-pycreds 0.4.0 pypi_0 pypi
docutils 0.16 pypi_0 pypi
einops 0.8.0 pypi_0 pypi
email-validator 2.2.0 pypi_0 pypi
evaluate 0.4.3 pypi_0 pypi
exceptiongroup 1.2.2 pypi_0 pypi
executing 2.1.0 pypi_0 pypi
fastapi 0.115.6 pypi_0 pypi
fasteners 0.19 pypi_0 pypi
fastjsonschema 2.21.1 pypi_0 pypi
fastrlock 0.8.3 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
ffmpy 0.5.0 pypi_0 pypi
filelock 3.16.1 pypi_0 pypi
flake8 7.1.0 pypi_0 pypi
flash-attn 2.7.2.post1 pypi_0 pypi
flask 3.1.0 pypi_0 pypi
flask-restful 0.3.10 pypi_0 pypi
fonttools 4.55.3 pypi_0 pypi
fqdn 1.5.1 pypi_0 pypi
freetype 2.12.1 h4a9f257_0
frozenlist 1.5.0 py310h5eee18b_0
fsspec 2024.3.1 py310h06a4308_0
genson 1.3.0 pypi_0 pypi
gflags 2.2.2 h6a678d5_1
gguf 0.10.0 pypi_0 pypi
giflib 5.2.2 h5eee18b_0
gitdb 4.0.12 pypi_0 pypi
gitpython 3.1.44 pypi_0 pypi
glog 0.5.0 h6a678d5_1
gmp 6.2.1 h295c915_3
gmpy2 2.1.2 py310heeb90bb_0
gnutls 3.6.15 he1e5248_0
gradio 5.9.1 pypi_0 pypi
gradio-client 1.5.2 pypi_0 pypi
grpcio 1.69.0 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
hiredis 3.1.0 pypi_0 pypi
hjson 3.1.0 pypi_0 pypi
httpcore 1.0.7 pypi_0 pypi
httptools 0.6.4 pypi_0 pypi
httpx 0.28.1 pypi_0 pypi
huggingface-hub 0.27.0 pypi_0 pypi
hydra-core 1.3.2 pypi_0 pypi
icu 73.1 h6a678d5_0
identify 2.6.5 pypi_0 pypi
idna 3.10 pypi_0 pypi
imageio 2.36.1 pypi_0 pypi
imageio-ffmpeg 0.5.1 pypi_0 pypi
importlib-metadata 8.5.0 pypi_0 pypi
inflect 5.6.2 pypi_0 pypi
iniconfig 2.0.0 pypi_0 pypi
intel-openmp 2023.1.0 hdb19cb5_46306
interegular 0.3.3 pypi_0 pypi
ipykernel 6.29.5 pypi_0 pypi
ipython 8.31.0 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
isort 5.13.2 pypi_0 pypi
itsdangerous 2.2.0 pypi_0 pypi
jedi 0.19.2 pypi_0 pypi
jinja2 3.1.5 pypi_0 pypi
jiter 0.8.2 pypi_0 pypi
jmespath 1.0.1 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
jpeg 9e h5eee18b_3
json5 0.10.0 pypi_0 pypi
jsonlines 4.0.0 pypi_0 pypi
jsonpointer 3.0.0 pypi_0 pypi
jsonschema 4.23.0 pypi_0 pypi
jsonschema-specifications 2024.10.1 pypi_0 pypi
jupyter-client 8.6.3 pypi_0 pypi
jupyter-core 5.7.2 pypi_0 pypi
jupyter-events 0.11.0 pypi_0 pypi
jupyter-lsp 2.2.5 pypi_0 pypi
jupyter-server 2.15.0 pypi_0 pypi
jupyter-server-terminals 0.5.3 pypi_0 pypi
jupyterlab 4.3.4 pypi_0 pypi
jupyterlab-pygments 0.3.0 pypi_0 pypi
jupyterlab-server 2.27.3 pypi_0 pypi
kiwisolver 1.4.8 pypi_0 pypi
krb5 1.20.1 h143b758_1
ks3sdk 1.11.0 pypi_0 pypi
lame 3.100 h7b6447c_0
lark 1.2.2 pypi_0 pypi
lazy-loader 0.4 pypi_0 pypi
lcms2 2.16 hb9589c4_0
ld_impl_linux-64 2.40 h12ee557_0
lerc 4.0.0 h6a678d5_0
libabseil 20240116.2 cxx17_h6a678d5_0
libboost 1.82.0 h109eef0_2
libbrotlicommon 1.0.9 h5eee18b_8
libbrotlidec 1.0.9 h5eee18b_8
libbrotlienc 1.0.9 h5eee18b_8
libcublas 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufile 1.11.1.6 0 nvidia
libcurand 10.3.7.77 0 nvidia
libcurl 8.11.1 hc9e6f67_0
libcusolver 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libdeflate 1.22 h5eee18b_0
libedit 3.1.20230828 h5eee18b_0
libev 4.33 h7f8727e_1
libevent 2.1.12 hdbd6064_1
libffi 3.4.4 h6a678d5_1
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libgrpc 1.62.2 h2d74bed_0
libiconv 1.16 h5eee18b_3
libidn2 2.3.4 h5eee18b_0
libjpeg-turbo 2.0.0 h9bf148f_0 pytorch
libnacl 2.1.0 pypi_0 pypi
libnghttp2 1.57.0 h2d74bed_0
libnpp 12.0.2.50 0 nvidia
libnvjitlink 12.1.105 0 nvidia
libnvjpeg 12.1.1.14 0 nvidia
libpng 1.6.39 h5eee18b_0
libprotobuf 4.25.3 he621ea3_0
libretranslatepy 2.1.1 pypi_0 pypi
librosa 0.10.2.post1 pypi_0 pypi
libssh2 1.11.1 h251f7ec_0
libstdcxx-ng 11.2.0 h1234567_1
libtasn1 4.19.0 h5eee18b_0
libthrift 0.15.0 h1795dd8_2
libtiff 4.5.1 hffd6297_1
libunistring 0.9.10 h27cfd23_0
libuuid 1.41.5 h5eee18b_0
libwebp 1.3.2 h11a3e52_0
libwebp-base 1.3.2 h5eee18b_1
llvm-openmp 14.0.6 h9e868ea_0
llvmlite 0.43.0 pypi_0 pypi
lm-eval 0.4.4 pypi_0 pypi
lm-format-enforcer 0.10.9 pypi_0 pypi
lxml 5.3.0 pypi_0 pypi
lz4-c 1.9.4 h6a678d5_1
markdown 3.7 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 3.0.2 pypi_0 pypi
matplotlib 3.10.0 pypi_0 pypi
matplotlib-inline 0.1.7 pypi_0 pypi
mbstrdecoder 1.1.3 pypi_0 pypi
mccabe 0.7.0 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
megatron-energon 0.0.0 pypi_0 pypi
mistral-common 1.5.1 pypi_0 pypi
mistune 3.1.0 pypi_0 pypi
mkl 2023.1.0 h213fc3f_46344
mkl-service 2.4.0 py310h5eee18b_1
mkl_fft 1.3.11 py310h5eee18b_0
mkl_random 1.2.8 py310h1128e8f_0
more-itertools 10.5.0 pypi_0 pypi
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpmath 1.3.0 py310h06a4308_0
msgpack 1.1.0 pypi_0 pypi
msgspec 0.19.0 pypi_0 pypi
multidict 6.1.0 py310h5eee18b_0
multiprocess 0.70.15 py310h06a4308_0
mypy 1.11.1 pypi_0 pypi
mypy-extensions 1.0.0 pypi_0 pypi
nbclient 0.10.2 pypi_0 pypi
nbconvert 7.16.5 pypi_0 pypi
nbformat 5.10.4 pypi_0 pypi
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 pypi_0 pypi
nettle 3.7.3 hbbd107a_1
networkx 3.4.2 pypi_0 pypi
ninja 1.11.1.3 pypi_0 pypi
nltk 3.9.1 pypi_0 pypi
nodeenv 1.9.1 pypi_0 pypi
notebook 7.3.2 pypi_0 pypi
notebook-shim 0.2.4 pypi_0 pypi
numba 0.60.0 pypi_0 pypi
numcodecs 0.13.1 pypi_0 pypi
numexpr 2.10.1 py310h3c60e43_0
numpy 2.2.1 pypi_0 pypi
numpy-base 1.26.4 py310hb5e798b_0
nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
nvidia-ml-py 12.560.30 pypi_0 pypi
nvidia-nccl-cu12 2.21.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
omegaconf 2.3.0 pypi_0 pypi
openai 1.59.3 pypi_0 pypi
opencv-python 4.10.0.84 pypi_0 pypi
opencv-python-headless 4.10.0.84 pypi_0 pypi
openh264 2.1.1 h4ff587b_0
openjpeg 2.5.2 he7f1fd0_0
openssl 3.0.15 h5eee18b_0
orc 2.0.1 h2d29ad5_0
orjson 3.10.13 pypi_0 pypi
outlines 0.0.46 pypi_0 pypi
overrides 7.7.0 pypi_0 pypi
packaging 24.2 py310h06a4308_0
pandas 2.2.3 py310h6a678d5_0
pandocfilters 1.5.1 pypi_0 pypi
parso 0.8.4 pypi_0 pypi
partial-json-parser 0.2.1.1.post4 pypi_0 pypi
pathspec 0.12.1 pypi_0 pypi
pathvalidate 3.2.3 pypi_0 pypi
peft 0.14.0 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pillow 10.4.0 pypi_0 pypi
pip 24.2 py310h06a4308_0
platformdirs 4.3.6 pypi_0 pypi
pluggy 1.5.0 pypi_0 pypi
pooch 1.8.2 pypi_0 pypi
portalocker 3.1.1 pypi_0 pypi
pre-commit 4.0.1 pypi_0 pypi
prometheus-client 0.21.1 pypi_0 pypi
prometheus-fastapi-instrumentator 7.0.0 pypi_0 pypi
prompt-toolkit 3.0.48 pypi_0 pypi
propcache 0.2.0 py310h5eee18b_0
protobuf 5.29.2 pypi_0 pypi
psutil 6.1.1 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.3 pypi_0 pypi
py 1.11.0 pypi_0 pypi
py-cpuinfo 9.0.0 pypi_0 pypi
pyairports 2.1.1 pypi_0 pypi
pyarrow 17.0.0 py310h1128e8f_0
pyasn1 0.6.1 pypi_0 pypi
pybind11 2.13.6 pypi_0 pypi
pycodestyle 2.12.1 pypi_0 pypi
pycountry 24.6.1 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pycryptodome 3.20.0 pypi_0 pypi
pydantic 2.10.4 pypi_0 pypi
pydantic-core 2.27.2 pypi_0 pypi
pydub 0.25.1 pypi_0 pypi
pyflakes 3.2.0 pypi_0 pypi
pygments 2.19.0 pypi_0 pypi
pylint 3.2.6 pypi_0 pypi
pyparsing 3.2.1 pypi_0 pypi
pysocks 1.7.1 py310h06a4308_0
pytablewriter 1.2.1 pypi_0 pypi
pytest 8.3.4 pypi_0 pypi
pytest-asyncio 0.25.1 pypi_0 pypi
pytest-cov 6.0.0 pypi_0 pypi
pytest-forked 1.6.0 pypi_0 pypi
pytest-mock 3.14.0 pypi_0 pypi
pytest-random-order 1.1.1 pypi_0 pypi
pytest-rerunfailures 15.0 pypi_0 pypi
pytest-shard 0.1.2 pypi_0 pypi
python 3.10.16 he870216_1
python-dateutil 2.9.0post0 py310h06a4308_2
python-dotenv 1.0.1 pypi_0 pypi
python-json-logger 3.2.1 pypi_0 pypi
python-multipart 0.0.20 pypi_0 pypi
python-tzdata 2023.3 pyhd3eb1b0_0
python-xxhash 2.0.2 py310h5eee18b_1
pytorch 2.5.1 py3.10_cuda12.1_cudnn9.1.0_0 pytorch
pytorch-cuda 12.1 ha16c6d3_6 pytorch
pytorch-mutex 1.0 cuda pytorch
pytz 2024.2 pypi_0 pypi
pyyaml 6.0.2 py310h5eee18b_0
pyzmq 26.2.0 pypi_0 pypi
ray 2.40.0 pypi_0 pypi
re2 2022.04.01 h295c915_0
readline 8.2 h5eee18b_0
redis 5.2.1 pypi_0 pypi
referencing 0.35.1 pypi_0 pypi
regex 2024.11.6 pypi_0 pypi
requests 2.32.3 py310h06a4308_1
rfc3339-validator 0.1.4 pypi_0 pypi
rfc3986-validator 0.1.1 pypi_0 pypi
rich 13.9.4 pypi_0 pypi
rouge-score 0.1.2 pypi_0 pypi
rpds-py 0.22.3 pypi_0 pypi
rsa 4.7.2 pypi_0 pypi
ruff 0.6.5 pypi_0 pypi
s2n 1.3.27 hdbd6064_0
s3fs 0.4.2 pypi_0 pypi
s3transfer 0.10.4 pypi_0 pypi
sacrebleu 2.5.1 pypi_0 pypi
safehttpx 0.1.6 pypi_0 pypi
safetensors 0.5.0 pypi_0 pypi
scikit-learn 1.6.0 pypi_0 pypi
scipy 1.15.0 pypi_0 pypi
semantic-version 2.10.0 pypi_0 pypi
send2trash 1.8.3 pypi_0 pypi
sentence-transformers 3.3.1 pypi_0 pypi
sentencepiece 0.2.0 pypi_0 pypi
sentry-sdk 2.19.2 pypi_0 pypi
setproctitle 1.3.4 pypi_0 pypi
setuptools 75.1.0 py310h06a4308_0
setuptools-scm 8.1.0 pypi_0 pypi
shellingham 1.5.4 pypi_0 pypi
six 1.17.0 pypi_0 pypi
smmap 5.0.2 pypi_0 pypi
snappy 1.2.1 h6a678d5_0
sniffio 1.3.1 pypi_0 pypi
soundfile 0.13.0 pypi_0 pypi
soupsieve 2.6 pypi_0 pypi
soxr 0.5.0.post1 pypi_0 pypi
sqlite 3.45.3 h5eee18b_0
sqlitedict 2.1.0 pypi_0 pypi
sr 0.1.0 pypi_0 pypi
stack-data 0.6.3 pypi_0 pypi
starlette 0.41.3 pypi_0 pypi
sympy 1.13.1 pypi_0 pypi
tabledata 1.3.4 pypi_0 pypi
tabulate 0.9.0 pypi_0 pypi
tbb 2021.8.0 hdb19cb5_0
tcolorpy 0.1.7 pypi_0 pypi
tenacity 9.0.0 pypi_0 pypi
tensorboard 2.18.0 pypi_0 pypi
tensorboard-data-server 0.7.2 pypi_0 pypi
tensorizer 2.9.1 pypi_0 pypi
tensorstore 0.1.45 pypi_0 pypi
terminado 0.18.1 pypi_0 pypi
threadpoolctl 3.5.0 pypi_0 pypi
tiktoken 0.7.0 pypi_0 pypi
timm 1.0.12 pypi_0 pypi
tinycss2 1.4.0 pypi_0 pypi
tk 8.6.14 h39e8969_0
tokenizers 0.21.0 pypi_0 pypi
toml 0.10.2 pypi_0 pypi
tomli 2.0.1 pypi_0 pypi
tomlkit 0.13.2 pypi_0 pypi
torchaudio 2.5.1 pypi_0 pypi
torchtriton 3.1.0 py310 pytorch
torchvision 0.20.1 pypi_0 pypi
tornado 6.4.2 pypi_0 pypi
tqdm 4.67.1 pypi_0 pypi
tqdm-multiprocess 0.0.11 pypi_0 pypi
traitlets 5.14.3 pypi_0 pypi
transformers 4.47.1 pypi_0 pypi
transformers-stream-generator 0.0.5 pypi_0 pypi
translate 3.6.1 pypi_0 pypi
triton 3.1.0 pypi_0 pypi
typepy 1.3.4 pypi_0 pypi
typer 0.15.1 pypi_0 pypi
types-python-dateutil 2.9.0.20241206 pypi_0 pypi
types-pyyaml 6.0.12.20241230 pypi_0 pypi
types-requests 2.32.0.20241016 pypi_0 pypi
types-setuptools 75.6.0.20241223 pypi_0 pypi
typing-extensions 4.12.2 py310h06a4308_0
typing_extensions 4.12.2 py310h06a4308_0
tzdata 2024.2 pypi_0 pypi
uri-template 1.3.0 pypi_0 pypi
urllib3 2.3.0 pypi_0 pypi
utf8proc 2.6.1 h5eee18b_1
uvicorn 0.34.0 pypi_0 pypi
uvloop 0.21.0 pypi_0 pypi
virtualenv 20.28.1 pypi_0 pypi
vllm 0.6.1.dev39+g0ba84350.d20250106 pypi_0 pypi
wandb 0.19.1 pypi_0 pypi
watchfiles 1.0.3 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
webcolors 24.11.1 pypi_0 pypi
webdataset 0.2.100 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.8.0 pypi_0 pypi
websockets 14.1 pypi_0 pypi
werkzeug 3.1.3 pypi_0 pypi
wheel 0.44.0 py310h06a4308_0
word2number 1.1 pypi_0 pypi
wrapt 1.17.0 pypi_0 pypi
xformers 0.0.28.post3 pypi_0 pypi
xgrammar 0.1.8 pypi_0 pypi
xxhash 0.8.0 h7f8727e_3
xz 5.4.6 h5eee18b_1
yaml 0.2.5 h7b6447c_0
yapf 0.32.0 pypi_0 pypi
yarl 1.18.0 py310h5eee18b_0
zarr 2.18.3 pypi_0 pypi
zipp 3.21.0 pypi_0 pypi
zlib 1.2.13 h5eee18b_1
zstandard 0.23.0 pypi_0 pypi
zstd 1.5.6 hc292b87_0
@zhaoyinglia
Copy link
Collaborator

zhaoyinglia commented Jan 7, 2025 •
edited
Loading

@lambert-x Hi, thank you for your issue. Could you provide your configuration about the llm_engine in inference_emu3_t2i.yaml? like:

llm:
  model: BAAI/Emu3-Gen
  vq_model: BAAI/Emu3-VisionTokenizer/
  skip_tokenizer_init: true
  trust_remote_code: true
  dtype: bfloat16
  tensor_parallel_size: 1
  pipeline_parallel_size: 1
  gpu_memory_utilization: 0.5
  disable_log_stats: false
  enable_chunked_prefill: false

@lambert-x
Copy link
Author

@lambert-x Hi, thank you for your issue. Could you provide your configuration about the llm_engine in inference_emu3_t2i.yaml? like:

llm:
  model: BAAI/Emu3-Gen
  vq_model: BAAI/Emu3-VisionTokenizer/
  skip_tokenizer_init: true
  trust_remote_code: true
  dtype: bfloat16
  tensor_parallel_size: 1
  pipeline_parallel_size: 1
  gpu_memory_utilization: 0.5
  disable_log_stats: false
  enable_chunked_prefill: false

Here is my config:

llm:
model: BAAI/Emu3-Gen
vq_model: BAAI/Emu3-VisionTokenizer/
skip_tokenizer_init: true
trust_remote_code: true
tensor_parallel_size: 1
pipeline_parallel_size: 1
gpu_memory_utilization: 0.5
disable_log_stats: false

generate:
mode: 'G'
prompts: [
"a portrait of young girl.",
"a shiba inu",
]
ratios: [
"1:1",
"16:9"
]
sampling:
top_k: 2048
max_tokens: 40960
guidance_scale: 3.0
detokenize: false

@zhaoyinglia
Copy link
Collaborator

zhaoyinglia commented Jan 7, 2025 •
edited
Loading

@lambert-x
It works fine for me with the latest version of FlagScale (commit id: 8ca54c6). You can try updating to the newest FlagScale and testing again. If the AssertionError persists, please provide additional details, such as:

  1. How you installed FS's vLLM.
  2. The exact command you are using.
  3. Add the following debug prints at codes/FlagScale/flagscale/inference/core/scheduler.py:1020:
print(f"{num_prompt_tokens=}")
print(f"{num_new_tokens=}")

@zhaoyinglia
Copy link
Collaborator

As a reference: output.log

@lambert-x
Copy link
Author

`#!/bin/bash

source /mnt/anaconda3/etc/profile.d/conda.sh && conda activate emu3
mkdir -p /mnt/codes/FlagScale/outputs/emu3_gen/inference_logs
mkdir -p /mnt/codes/FlagScale/outputs/emu3_gen/inference_logs/pids

cd /mnt/codes/FlagScale

export PYTHONPATH=/mnt/codes/FlagScale/vllm:/mnt/codes/FlagScale

cmd="CUDA_VISIBLE_DEVICES=4 CUDA_DEVICE_MAX_CONNECTIONS=1 python flagscale/inference/inference_emu3.py --config-path=/mnt/codes/FlagScale/outputs/emu3_gen/hydra/.hydra/config.yaml"

nohup bash -c "$cmd; sync" >> /mnt/codes/FlagScale/outputs/emu3_gen/inference_logs/host_0_localhost.output 2>&1 & echo $! > /mnt/codes/FlagScale/outputs/emu3_gen/inference_logs/pids/host_0_localhost.pid

`

The compiled script is provided above. If I directly run this, I will get the error message of :

Processed prompts: 0%| | 0/2 [01:51<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s] [rank0]:[W106 20:09:12.119509578 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) Traceback (most recent call last): File "/mnt/codes/FlagScale/flagscale/inference/inference_emu3.py", line 13, in <module> from vllm import LLM File "/mnt/codes/FlagScale/vllm/vllm/__init__.py", line 3, in <module> from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs File "/mnt/codes/FlagScale/vllm/vllm/engine/arg_utils.py", line 11, in <module> from vllm.config import (CacheConfig, CompilationConfig, ConfigFormat, File "/mnt/codes/FlagScale/vllm/vllm/config.py", line 20, in <module> from vllm.model_executor.layers.quantization import (QUANTIZATION_METHODS, File "/mnt/codes/FlagScale/vllm/vllm/model_executor/__init__.py", line 1, in <module> from vllm.model_executor.parameter import (BasevLLMParameter, File "/mnt/codes/FlagScale/vllm/vllm/model_executor/parameter.py", line 7, in <module> from vllm.distributed import get_tensor_model_parallel_rank File "/mnt/codes/FlagScale/vllm/vllm/distributed/__init__.py", line 1, in <module> from .communication_op import * File "/mnt/codes/FlagScale/vllm/vllm/distributed/communication_op.py", line 6, in <module> from .parallel_state import get_tp_group File "/mnt/codes/FlagScale/vllm/vllm/distributed/parallel_state.py", line 38, in <module> import vllm.distributed.kv_transfer.kv_transfer_agent as kv_transfer File "/mnt/codes/FlagScale/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 15, in <module> from vllm.distributed.kv_transfer.kv_connector.factory import ( File "/mnt/codes/FlagScale/vllm/vllm/distributed/kv_transfer/kv_connector/factory.py", line 3, in <module> from .base import KVConnectorBase File "/mnt/codes/FlagScale/vllm/vllm/distributed/kv_transfer/kv_connector/base.py", line 14, in <module> from vllm.sequence import IntermediateTensors File "/mnt/codes/FlagScale/flagscale/utils.py", line 43, in load_module spec.loader.exec_module(module) File "/mnt/codes/FlagScale/flagscale/inference/core/sequence.py", line 17, in <module> from vllm.inputs import SingletonInputs, SingletonInputsAdapter File "/mnt/codes/FlagScale/vllm/vllm/inputs/__init__.py", line 7, in <module> from .registry import (DummyData, InputContext, InputProcessingContext, File "/mnt/codes/FlagScale/vllm/vllm/inputs/registry.py", line 13, in <module> from vllm.transformers_utils.tokenizer import AnyTokenizer File "/mnt/codes/FlagScale/vllm/vllm/transformers_utils/tokenizer.py", line 16, in <module> from vllm.utils import make_async File "/mnt/codes/FlagScale/vllm/vllm/utils.py", line 44, in <module> from vllm.platforms import current_platform File "/mnt/codes/FlagScale/vllm/vllm/platforms/__init__.py", line 100, in <module> from .cuda import CudaPlatform File "/mnt/codes/FlagScale/vllm/vllm/platforms/cuda.py", line 14, in <module> import vllm._C # noqa ModuleNotFoundError: No module named 'vllm._C'

@lambert-x
Copy link
Author

How should I install the vllm? I actually follow the readme but it seems the environment gets confused with the vllm path to be used.

@zhaoyinglia
Copy link
Collaborator

zhaoyinglia commented Jan 7, 2025 •
edited
Loading

@lambert-x Here’s the method I used to install vLLM, feel free to give it a try!

conda activate emu3
cd FlagScale/vllm
pip install --no-build-isolation -v -e .

@zhaoyinglia
Copy link
Collaborator

Hi @lambert-x, did the method above work for you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lambert-x @zhaoyinglia