Qlora with flan-t5 issue - ValueError: Trying to set a tensor of shape torch.Size([4096, 4096])

[JhonDan1999]

Hi,

I want to use Qlora with Flan-t5

but I got this error

ValueError: Trying to set a tensor of shape torch.Size([4096, 4096]) in "weight" (which has shape torch.Size([8388608, 1])), this look incorrect.

here is my code

model_name = "google/flan-t5-xxl"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type='nf4',
    bnb_4bit_compute_dtype=torch.bfloat16,

    bnb_4bit_use_double_quant=True,
    llm_int8_enable_fp32_cpu_offload=True
)

model = AutoModelForSeq2SeqLM.from_pretrained( #for "google/flan-t5-xxl"
    model_name,
    quantization_config=bnb_config,
    device_map='auto',
    torch_dtype=torch.bfloat16,
)
model.config.use_cache = False
model.config.pretraining_tp = 1
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

the error occur earn I call the generate here

    # Generate model output using greedy decoding
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=1, do_sample=False,top_p=None)

can you please help me with this