Support for PEFT and BitsAndBytesConfig with Liger Kernel? If not, any advise for quantization and lora?

[GianottiGustavo]

Hi everyone,

I have been trying to use PEFT with BitsAndBytesConfig iwith AutoLigerKernelForCausalLM.from_pretrained. However, I keep encountering type errors during training.
"RuntimeError: self and mat2 must have the same dtype, but got Half and Float"

Could anyone clarify if PEFT and BitsAndBytesConfig are officially supported with the Liger kernel? If they are, could you provide any specific guidance or documentation references that might help resolve these type errors?
If not, any advise for quantization and lora?

THX!

I tried in several ways; here is just one example to show:

model = AutoLigerKernelForCausalLM.from_pretrained("PrunaAI/Qwen-Qwen2-0.5B-Instruct-bnb-4bit-smashed", 
                                                   low_cpu_mem_usage=True, device_map={"":0})

lora_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0,
    r=128,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "down_proj",
        "up_proj",
    ],
    use_rslora=False,
    loftq_config=None,
)

model = get_peft_model(model, lora_config)

sft_config = SFTConfig(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=5,
        # num_train_epochs=1,  # We manage epochs manually
        max_steps=30,  # Add chunk size to the current step
        learning_rate=2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps=50,
        optim="adamw_8bit", # grokadamw, pip install grokadamw. other is paged_adamw_8bit.
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir=folder_model_path + "outputs",
        logging_dir=folder_model_path + "logs",
        save_strategy="steps",
        save_steps=50,
        max_seq_length=max_seq_length,
        dataset_text_field="text",
        dataset_num_proc=2,
    )
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    packing=False,  # Can make training faster for short sequences.
    args=sft_config,
    peft_config=lora_config,
)
trainer_stats = trainer.train()

[ByronHsu]

Hi can you create a ticket instead? Thanks!