You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The paper shows that you quantize the weights to 2/4 bits using NF format. I wonder how to you handle the input activations (denoted as x). Is x also quantized to 2/4 bits?
If not, do you dequantize the quantized weights to float format to perform the matmul using float kernels?
This seems to slow down the inference efficiency despite lower memory footprint.
The text was updated successfully, but these errors were encountered:
The paper "QLORA: Efficient Finetuning of Quantized LLMs" says "In practice, this means whenever a QLORA weight tensor is used, we dequantize the tensor to BFloat16, and then perform a matrix multiplication in 16-bit." As far as i'm concerned, NF16 and Double Quantization may slow down the training and inference efficiency, which means this work sacrifices time and performance for memory. However, this paper reports "24hours of finetuning on a single GPU", and I wonder how.
hi @TimDettmers .
The paper shows that you quantize the weights to 2/4 bits using NF format. I wonder how to you handle the input activations (denoted as x). Is x also quantized to 2/4 bits?
If not, do you dequantize the quantized weights to float format to perform the matmul using float kernels?
This seems to slow down the inference efficiency despite lower memory footprint.
The text was updated successfully, but these errors were encountered: