[Bug]: InternVL bounding box prediction does not work #9153
Comments
|
Currently, we only support inserting image tokens ( |
|
@ywang96 together with #7905 (comment), this is another case where we need to override how the image tokens are inserted per image input. We may have to work on this soon. |
|
This is odd, because However, both |
|
Thanks for your quick responses! @DarkLight1337 , I'm not sure if this really requires a special logic/implementation. The prompt @Isotr0py , that's very interesting. I had only tried InternVL-1B, because deploying 2B gave me this error (although 1B had accepted DISABLE_SLIDING_WINDOW=true): Exit code: 1. Reason: ^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 207, in __init__
self.max_model_len = _get_and_verify_max_len(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1738, in _get_and_verify_max_len
raise NotImplementedError(
NotImplementedError: Disabling sliding window is not supported for models with rope_scaling. Please raise an issue so we can investigate.
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 388, in run_mp_engine
engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 134, in from_engine_args
engine_config = engine_args.create_engine_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 874, in create_engine_config
model_config = self.create_model_config()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 811, in create_model_config
return ModelConfig(
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 207, in __init__
self.max_model_len = _get_and_verify_max_len(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1738, in _get_and_verify_max_len
raise NotImplementedError(
NotImplementedError: Disabling sliding window is not supported for models with rope_scaling. Please raise an issue so we can investigate. |
|
Interesting, I confirm that I also get bounding boxes with the 2B model. (when successfully deployed with DISABLE_SLIDING_WINDOW=false) |
|
I tested whether the issue comes from DISABLE_SLIDING_WINDOW, but I still get gibberish with the 1B model when changing to DISABLE_SLIDING_WINDOW=false. (I noticed that internvl tokenizes images to many tokens (3k+ tokens for just one image and a short text prompt), so thought that it could maybe be linked to the sliding window making the model only pay attention to parts of the input) |
|
Seems that |
|
I also just got it to work with the 4b model |
|
I have tryied to use OpenGVLab/InternVL2-Llama3-76B-AWQ for bbox prediction. I use image 1920x1080. It correctly answers in json format, but bbox coordinates are bad. |
|
Description: I am experiencing an issue with the VLLM + InternVL models when attempting to extract titles from images （OCR task）. Specifically:
Steps to Reproduce:
Expected Outcome: The 1B model should provide results consistent with the Transformers model and the 8B model. Actual Outcome: The 1B model in the VLLM + InternVL setup produces incorrect results. |
Can you provide an example of how you passed the prompt text and image to the model? |
|
ok thank you , here is my code: use vllmfrom vllm import LLM, SamplingParams
from PIL import Image
llm = LLM(model="/data/OpenGVLab/InternVL2-1B", max_model_len=2048, gpu_memory_utilization=0.3, max_num_seqs=1, trust_remote_code=True, )
# sampling_params = SamplingParams(temperature=1, max_tokens=128)
sampling_params = SamplingParams(
best_of=1,
temperature=1.0,
top_p=1,
top_k=1,
# use_beam_search=False,
max_tokens=128,
presence_penalty=0,
frequency_penalty=0,
)
fp = "/data/images/0031.jpg"
image = Image.open(fp)
outputs = llm.generate({
"prompt": "USER: <image>\n图片中的标题是什么？\nASSISTANT:",
"multi_modal_data": {"image": image},
}, sampling_params
)
print("~" * 30)
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)and it output nothing use huggingface transformers# copied from https://huggingface.co/OpenGVLab/InternVL2-8B#inference-with-transformers
path = '/data/OpenGVLab/InternVL2-1B'
model = AutoModel.from_pretrained(
path,
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
use_flash_attn=True,
trust_remote_code=True).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
# set the max number of tiles in `max_num`
fp = "/data/images/0031.jpg"
pixel_values = load_image(fp, max_num=12).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens=1024, do_sample=False)
question = '<image>\n图片中的标题是什么？'
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}\nAssistant: {response}')and it output is right : User: this is the subimage of my input image: with vllm, i tried some other SamplingParams, they were all failed. |
|
@aixuedegege Your prompt format used in vLLM is wrong, you can refer to examples/offline_inference_vision_language.py for chat template application. |
|
got it. thank you so much. |
Your current environment
The output of `python collect_env.py`
```text python collect_env.pyCollecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31
Python version: 3.9.5 (default, Jun 4 2021, 12:28:51) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.5.82
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A10G
Nvidia driver version: 535.161.07
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.2.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.2.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 48
On-line CPU(s) list: 0-47
Thread(s) per core: 2
Core(s) per socket: 24
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 23
Model: 49
Model name: AMD EPYC 7R32
Stepping: 0
CPU MHz: 3110.497
BogoMIPS: 5599.58
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 768 KiB
L1i cache: 768 KiB
L2 cache: 12 MiB
L3 cache: 96 MiB
NUMA node0 CPU(s): 0-47
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid
Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.77
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.2
[pip3] triton==3.0.0
[conda] numpy 1.26.4 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
[conda] nvidia-ml-py 12.560.30 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.6.77 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
[conda] pyzmq 26.2.0 pypi_0 pypi
[conda] torch 2.4.0 pypi_0 pypi
[conda] torchvision 0.19.0 pypi_0 pypi
[conda] transformers 4.45.2 pypi_0 pypi
[conda] triton 3.0.0 pypi_0 pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1.dev238+ge2c6e0a82
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 0-47 0 N/A
Legend:
X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks```
Model Input Dumps
No response
🐛 Describe the bug
I'm starting a vllm container with OpenGVLab/InternVL2 and vLLM version 0.6.2 with
vllm serve OpenGVLab/InternVL2-1B --trust-remote-codeAs a prompt, I'm specifically using the object detection prompt:
Please provide the bounding box coordinates of the region this sentence describes: <ref>{}</ref>. This prompt is documented here, here, or here. This prompt is designed to make the model output bounding box coordinates.This prompt works correctly when used in
transformers:When I use the same standard bounding box prompt with vLLM and the OAI client, however, the output is gibberish:
The expected output would be something like
the bird[[123, 234, 123, 234]]with bounding box coordinates for the object specified in the prompt.The vLLM container does successfully generate text when prompted without this bounding-box prompt. e.g. the prompt "What type of animal is in the image above? What color does it have?" leads to this output with this image:
The animal in the image is a species of bird. The bird in the image is a Blue Tit. The bird has a distinctive and colorful plumage, with a blue head and a yellow and a bit of a red or orange throat.I have tested this with several images and prompt variants and generation parameters. The bounding box prompt always outputs some form of gibberish, while other prompts output correct text. I also noticed that non-bounding box prompts produce correct text, but it also seems a bit suboptimal with repetitions and small errors.
I have a suspicion that there is something wrong in the chat template application done by vLLM for
OpenGVLab/InternVL2. My assumption is that, because the model has only been trained with this bounding box prompt with the very specific chat template, it fully fails when there is a small deviation from the chat template. For normal prompts, the negative effect of the chat template issue might be less visible, because it probably has seen a wider variety of input formats, so it still produces correct, but suboptimal outputs. But this is just an assumption and the issue might come from somewhere else. Optimal InternVL outputs can also be created via this official HF Space demo for comparison.I found the PR that added InternVL to vLLM, but I'm not sure which part of the code implements the preprocessor with the chat template.
Before submitting a new issue...
The text was updated successfully, but these errors were encountered: