vllm加速phi3v推理有bug #1398
Comments
|
有没有大佬知道怎么回事，今天配了一天环境确实解决不了这个问题TypeError: init() got an unexpected keyword argument 'image_input_type' |
|
这是个忧伤的问题 我帮你解决一下 |
|
vllm在0.5.1版本对多模态模型的推理进行了重构，导致swift对vllm多模态推理的支持只包括0.5.0.* 你可以使用vllm-project/vllm#6089 这个PR之前的vllm版本（但又需要在支持phi3-vision的PR之后）来使用swift支持phi3-vision |
|
swift对phi3-vision的支持 可以使用多图... |
谢谢你高速的回复！据我所知，v0.5.0.post1是swift支持的最高版本vllm，但是对phi3v的支持最早要等到v0.5.1了，也就是现阶段没法用vllm加速推理，对吗 |
|
已经支持vllm & vlm. 拉取一下main分支 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
What the bug is, and how to reproduce, better with screenshots(描述bug以及复现过程，最好有截图)
使用0.5.0的VLLM加速推理phi3v会有如下报错：[rank0]: ValueError: Model architectures ['Phi3VForCausalLM'] are not supported for now. Supported architectures: ['AquilaModel', 'AquilaForCausalLM', 'BaiChuanForCausalLM', 'BaichuanForCausalLM', 'BloomForCausalLM', 'ChatGLMModel', 'ChatGLMForConditionalGeneration', 'CohereForCausalLM', 'DbrxForCausalLM', 'DeciLMForCausalLM', 'DeepseekForCausalLM', 'FalconForCausalLM', 'GemmaForCausalLM', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GPTJForCausalLM', 'GPTNeoXForCausalLM', 'InternLMForCausalLM', 'InternLM2ForCausalLM', 'JAISLMHeadModel', 'LlamaForCausalLM', 'LlavaForConditionalGeneration', 'LlavaNextForConditionalGeneration', 'LLaMAForCausalLM', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MptForCausalLM', 'MPTForCausalLM', 'MiniCPMForCausalLM', 'OlmoForCausalLM', 'OPTForCausalLM', 'OrionForCausalLM', 'PhiForCausalLM', 'Phi3ForCausalLM', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2MoeForCausalLM', 'RWForCausalLM', 'StableLMEpochForCausalLM', 'StableLmForCausalLM', 'Starcoder2ForCausalLM', 'ArcticForCausalLM', 'XverseForCausalLM', 'Phi3SmallForCausalLM', 'MistralModel']
若使用0.5.1，又会出现transformer版本冲突。phi3v需求的是4.40.2，但是vllm0.5.1需求的是4.42.4
如果强行这样用就会报错：
FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use
slow_image_processor_class, orfast_image_processor_classinsteadwarnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Traceback (most recent call last):
File "/data1/hmy/generatechart/swift/swift/cli/infer.py", line 5, in
infer_main()
File "/data1/hmy/generatechart/swift/swift/utils/run_utils.py", line 27, in x_main
result = llm_x(args, **kwargs)
File "/data1/hmy/generatechart/swift/swift/llm/infer.py", line 270, in llm_infer
llm_engine, template = prepare_vllm_engine_template(args)
File "/data1/hmy/generatechart/swift/swift/llm/utils/vllm_utils.py", line 521, in prepare_vllm_engine_template
llm_engine = get_vllm_engine(
File "/data1/hmy/generatechart/swift/swift/llm/utils/vllm_utils.py", line 99, in get_vllm_engine
engine_args = engine_args_cls(
TypeError: init() got an unexpected keyword argument 'image_input_type'
Your hardware and system info
Write your system info like CUDA version/system/GPU/torch version here(在这里给出硬件信息和系统信息，如CUDA版本，系统，GPU型号和torch版本等)
Additional context
Add any other context about the problem here(在这里补充其他信息)
The text was updated successfully, but these errors were encountered: