support llava1.5 lora finetuning.

[lkk12014402]

What does this PR do?

add llava1.5 finetuning and add an example

[lkk12014402]

performance comparison

finetuning on gaudi

Before optimization

After optimization (this pr)

finetuning on a100

[yao-matrix]

@libinta , pls help review, validated pass in 1.19.0-410 build, thx.

[github-actions]

The code quality check failed, please run make style.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yafshar]

@lkk12014402 thanks for your contribution.

Can you explain why you created a new script, run_llava_lora_finetune.py, which is very similar to run_image2text_lora_finetune.py, instead of modifying the original script? Introducing new code can increase the possibility of errors and make maintenance more challenging.

[yafshar]

@lkk12014402 in your performance comparison above, what kind of optimization you did? I do not see any optimization. Can you elaborate?

[lkk12014402]

@lkk12014402 thanks for your contribution.

Can you explain why you created a new script, run_llava_lora_finetune.py, which is very similar to run_image2text_lora_finetune.py, instead of modifying the original script? Introducing new code can increase the possibility of errors and make maintenance more challenging.

hi, llava's Processor, DataCollator, evaluation code, padding style are different from the run_image2text_lora_finetune.py. If we merge the llava's code into the file run_image2text_lora_finetune.py, some if-else code will be ingested into the file. And I think each model maybe have its own Processor and DataCollator, or suitable dataset, So it is clear to create a new script.

[lkk12014402]

@lkk12014402 in your performance comparison above, what kind of optimization you did? I do not see any optimization. Can you elaborate?

padding inputs for static shape during training datacollator

[yafshar]

@lkk12014402 can you also add the tests results? Does your change breaks any unit tests (test_image_to_text_example)?

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s -k llava

before and after your changes?

[yafshar]

I ran the test

main

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s -k llava

10 passed, 4 deselected in 4419.67s (1:13:39)

this PR

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s -k llava

10 passed, 4 deselected in 2320.45s (0:38:40)

[yafshar]

I am still not sure about adding a new script but other than that it LGTM!

@regisss would you please check this PR.

[lkk12014402]

I ran the test

main

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s -k llava

10 passed, 4 deselected in 4419.67s (1:13:39)

this PR

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s -k llava

10 passed, 4 deselected in 2320.45s (0:38:40)

Thanks~ @yafshar I will merge llava run_llava_lora_finetune.py into the run_image2text_lora_finetune.py

[lkk12014402]

I am still not sure about adding a new script but other than that it LGTM!

@regisss would you please check this PR.

hi, @yafshar I have merged the 2 scripts.

Thanks~

[yafshar]

@lkk12014402 thanks. Can you do the full test to make sure your changes did not break anything?

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s 

[yafshar]

OK, I ran the tests. They seem to be fine. Would you please try some other examples

main branch

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s 
14 passed in 1957.63s (0:32:37)

this PR

>>> RUN_SLOW=true GAUDI2_CI=1 python -m pytest tests/test_image_to_text_example.py -v -s 
14 passed in 1428.22s (0:23:48)

[lkk12014402]

hi, @regisss please review~

[vidyasiv]

based on regiss feedback: please add/update relevant test(s) for new script/model: https://github.com/huggingface/optimum-habana/blob/main/tests/test_image_to_text_example.py