Gptq/exllama integration.

[nivibilla]

Hey, I know it's on the roadmap but if we are just talking about a single GPU solution to a small model(7b). How can I go about loading a model in 4 bit. Or even try to integrate exllama in here

Thanks