0.0.19

• More accurate Q4 cache using groupwise rotations
• Better prompt ingestion speed when using flash-attn
• Minor fixes related to issues quantizing Llama 3
• New, more robust optimizer
• Fix bug on long-sequence inference for GPTQ models

Full Changelog: v0.0.18...v0.0.19

0.0.18

• Support for Command-R-plus
• Fix for pre-AVX2 CPUs
• VRAM optimizations for quantization
• Very preliminary multimodal support
• Various other small fixes and optimizations

Full Changelog: v0.0.17...v0.0.18

0.0.17

Mostly just minor fixes and support for DBRX models.

Full Changelog: v0.0.16...v0.0.17

0.0.16

• Adds support for Cohere models
• N-gram decoding
• A few bugfixes
• Lots of optimizations

Full Changelog: v0.0.15...v0.0.16

0.0.15

• Adds Q4 cache mode
• Support for StarCoder2
• Minor optimizations and a couple of bugfixes

Full Changelog: v0.0.14...v0.0.15

0.0.14

Adds support for Qwen1.5 and Gemma architectures.

Various fixes and optimizations.

Full Changelog since 0.0.13: v0.0.13...v0.0.14

0.0.13.post2

Full Changelog: 0.0.13.post1...0.0.13.post2

0.0.13.post1

Fixes inference on models with vocab sizes that are not multiples of 32

0.0.13

This release is mostly to update the prebuilt wheels to Torch 2.2, since it won't load extensions built for earlier versions.

Adds dynamic temperature and quadratic sampling. Fixes performance degradation on some GPUs after batch optimizations and various other little things.

0.0.12

Lots of fixes and tweaks. Main feature updates:

Model support:

• Basic LoRA support for MoE models
• Support for Orion models (also groundwork for other layernorm models)
• Support for loading/converting from Axolotl checkpoints

Generation/sampling:

• Fused kernels enabled for num_experts = 4
• Option to return probs from streaming generator
• Add top-A sampling
• Add freq/pres penalties
• CFG support in streaming generator
• Disable flash-attn for non-causal attention (fixes left-padding until FA2 implements custom bias)

Testing/evaluation:

• HumanEval test
• Script to compare two models layer by layer (e.g. quantized vs. original model)
• "Standard" ppl test that attempts to mimic text-generation-webui

Conversion:

• VRAM optimizations
• Optimized quantization kernels

IO:

• Cache safetensors context managers for faster loading
• Optional direct IO loader (for very fast arrays)