Skip to content

Latest commit

 

History

History
30 lines (19 loc) · 1023 Bytes

README.md

File metadata and controls

30 lines (19 loc) · 1023 Bytes

InstructBLIP Finetune

This is the finetune of InstructBLIP paper.

Install

git clone https://github.com/ZhanKunLiAuto/LAVIS-AD
cd LAVIS-AD
pip install -e .

Prepare Vicuna Weights

InstructBLIP uses frozen 13B models. Please first follow the instructions to prepare Vicuna v1.1 weights. Then modify the model.llm_model in the lavis/configs/models/blip2/blip2_instruct_vicuna13b_autonomous_driving.yaml to the folder that contains Vicuna weights.

Train

python train.py --cfg-path lavis/configs/models/blip2/blip2_instruct_vicuna13b_autonomous_driving.yaml

Demo

  1. Modify the model.llm_model in the lavis/configs/models/blip2/blip2_instruct_vicuna13b.yaml to the folder that contains Vicuna weights.

  2. Modify the model.pretrained in the lavis/configs/models/blip2/blip2_instruct_vicuna13b.yaml to the checkpoint.

  3. Run Gradio Demo locally with python run_demo.py