[Model][LoRA]LoRA support added for MolmoForCausalLM

[ayylemao]

[Model][LoRA]LoRA support added for MolmoForCausalLM
LoRA supported for inference in MolmoForCausalLM

Need some help with verifying this works correctly since my own tests revealed the same results for base_model and base_model+ lora.

ping @jeejeelee

FIX #11431

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[jeejeelee]

Need some help with verifying this works correctly since my own tests revealed the same results for base_model and base_model+ lora.

I roughly know what the reason is. I will resolve it today, and we also need to update the documentation,see :https://docs.vllm.ai/en/latest/models/supported_models.html#id3

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ayylemao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[ayylemao]

i might have done goofed by syncing my fork and pushing to the PR branch
I fixed what i could, sorry for all the review requests...

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ayylemao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jeejeelee]

@ayylemao could you plz address the branch conflict ?

[ayylemao]

Thank you for your review.

I will get back to it after christmas!

[jeejeelee]

@ayylemao I've added modules with LoRA support. Could you test if it generates reasonable results?

[ayylemao]

@ayylemao I've added modules with LoRA support. Could you test if it generates reasonable results?

@jeejeelee After testing i still get the same results for the base model and base_model+lora when serving via vllm serve.
I think my lora adapter still does not get applied correctly, even though when evaluating via transformers+peft it works as expected.

I am not sure how to test more accuratly if the lora adapters get correctly applied or not beyond just comparing the actual model output.
Do you have any idea what might be the issue or how to pinpoint further?

[jeejeelee]

@ayylemao I've added modules with LoRA support. Could you test if it generates reasonable results?

@jeejeelee After testing i still get the same results for the base model and base_model+lora when serving via vllm serve. I think my lora adapter still does not get applied correctly, even though when evaluating via transformers+peft it works as expected.

I am not sure how to test more accuratly if the lora adapters get correctly applied or not beyond just comparing the actual model output. Do you have any idea what might be the issue or how to pinpoint further?

I'm training a LoRA model for this model and will test it tomorrow.

[jeejeelee]

@ayylemao I've added modules with LoRA support. Could you test if it generates reasonable results?

@jeejeelee After testing i still get the same results for the base model and base_model+lora when serving via vllm serve. I think my lora adapter still does not get applied correctly, even though when evaluating via transformers+peft it works as expected.
I am not sure how to test more accuratly if the lora adapters get correctly applied or not beyond just comparing the actual model output. Do you have any idea what might be the issue or how to pinpoint further?

I'm training a LoRA model for this model and will test it tomorrow.

@ayylemao After merging with #11551, when testing 100 samples, 99 generated results matched exactly with the transformers output. The remaining one also produced a reasonable result.

[ayylemao]

@ayylemao I've added modules with LoRA support. Could you test if it generates reasonable results?

@jeejeelee After testing i still get the same results for the base model and base_model+lora when serving via vllm serve. I think my lora adapter still does not get applied correctly, even though when evaluating via transformers+peft it works as expected.
I am not sure how to test more accuratly if the lora adapters get correctly applied or not beyond just comparing the actual model output. Do you have any idea what might be the issue or how to pinpoint further?

I'm training a LoRA model for this model and will test it tomorrow.

@ayylemao After merging with #11551, when testing 100 samples, 99 generated results matched exactly with the transformers output. The remaining one also produced a reasonable result.

Thank you for testing.

After your merge with #11551 i also get correct results!
Thank you very much for your help with this PR.

[jeejeelee]

@ayylemao Thanks for your contribution

[ayylemao]

Do i still need to do something here because of the failing buildkite/ci/pr/lora-test-1 check?
It seems that a Qwen2-VL check has failed.

[2024-12-30T10:24:42Z] =================================== FAILURES ===================================
--
  | [2024-12-30T10:24:42Z] ______________________________ test_qwen2vl_lora _______________________________
  | [2024-12-30T10:24:42Z]
  | [2024-12-30T10:24:42Z] qwen2vl_lora_files = '/root/.cache/huggingface/hub/models--jeeejeee--qwen2-vl-lora-pokemon/snapshots/6c85687748763f7449fa15c345520b43ba6e522f'
  | [2024-12-30T10:24:42Z]
  | [2024-12-30T10:24:42Z]     @pytest.mark.xfail(current_platform.is_rocm(),
  | [2024-12-30T10:24:42Z]                        reason="Qwen2-VL dependency xformers incompatible with ROCm"
  | [2024-12-30T10:24:42Z]                        )
  | [2024-12-30T10:24:42Z]     def test_qwen2vl_lora(qwen2vl_lora_files):
  | [2024-12-30T10:24:42Z]         llm = vllm.LLM(
  | [2024-12-30T10:24:42Z]             MODEL_PATH,
  | [2024-12-30T10:24:42Z]             max_num_seqs=2,
  | [2024-12-30T10:24:42Z]             enable_lora=True,
  | [2024-12-30T10:24:42Z]             max_loras=2,
  | [2024-12-30T10:24:42Z]             max_lora_rank=16,
  | [2024-12-30T10:24:42Z]             trust_remote_code=True,
  | [2024-12-30T10:24:42Z]             mm_processor_kwargs={
  | [2024-12-30T10:24:42Z]                 "min_pixels": 28 * 28,
  | [2024-12-30T10:24:42Z]                 "max_pixels": 1280 * 28 * 28,
  | [2024-12-30T10:24:42Z]             },
  | [2024-12-30T10:24:42Z]             max_model_len=4096,
  | [2024-12-30T10:24:42Z]         )
  | [2024-12-30T10:24:42Z]         output1 = do_sample(llm, qwen2vl_lora_files, lora_id=1)
  | [2024-12-30T10:24:42Z]         for i in range(len(EXPECTED_OUTPUT)):
  | [2024-12-30T10:24:42Z] >           assert EXPECTED_OUTPUT[i].startswith(output1[i])
  | [2024-12-30T10:24:42Z] E           AssertionError: assert False
  | [2024-12-30T10:24:42Z] E            +  where False = <built-in method startswith of str object at 0x7f462c0639f0>('A vibrant street scene with')
  | [2024-12-30T10:24:42Z] E            +    where <built-in method startswith of str object at 0x7f462c0639f0> = 'A red stop sign stands prominently in the foreground, with a traditional Chinese gate and a black SUV in the background, illustrating a blend of modern and cultural elements.'.startswith
  | [2024-12-30T10:24:42Z]
  | [2024-12-30T10:24:42Z] lora/test_qwen2vl.py:78: AssertionError

[jeejeelee]

@ayylemao This is a known issue and is not related to this PR.