Support GGUF

[philpax]

GGUF is the new file format specification that we've been designing that's designed to solve the problem of not being able to identify a model. The specification is here: ggerganov/ggml#302

llm should be able to do the following:

• continue supporting existing models (i.e. this change should be non-destructive)
• load GGUF models and automatically dispatch to the correct model.
  • load_dynamic already has an interface that should support this, but loading currently only begins after the model arch is known
  • use the new information stored within the metadata to improve the UX, including automatically using the HF tokenizer if available
• save GGUF models, especially in quantization

llm could do the following:

• convert old models to GGUF models with prompting for missing data
• implement the migration tool mentioned in the spec, which does autonomous conversion for users based on hashes

[EwoutH]

To give an update on the state of GGUF: Halfway August GGUF was merged into llama.cpp (ggerganov/llama.cpp#2398 (comment)). It’s full specification can be found here.

Recap of what GGUF is:

• binary file format for storing models for inference
• designed for fast loading and saving of models
• easy to use (with a few lines of code)
• mmap (memory mapping) compatibility: models can be loaded using mmap for fast loading and saving.

[pixelspark]

Hi all, any updates on this?

[philpax]

Hi - sorry about the lack of updates, I've been extremely busy for the last ~two months and haven't had much free time to work on llm. I'm hoping this will ease up soon and we can start catching up proper.

[Dipeshpal]

Any updates on gguf?