Skip to content

Kobold.CPP_Frankenstein_v1.67d_b3075_KaViar_Edition
Compare
Choose a tag to compare
@Nexesenex Nexesenex released this 03 Jun 23:34
· 39 commits to frankenknives since this release
v1.67d_b3075
92cdfcd

Frankenstein 1.67d "Fork" of KoboldCPP Experimental up to the 3/06/2024, 14h GMT+2.
Based on Llama.CPP b3075.

Disclaimer:

These third party KoboldCPP-Frankenstein builds are not supported in any way by the KoboldCPP team, Github, or discord channel. They are for greedy-test and amusement only.
My KCPP-Frankenstein version number bumps as soon as the version number in the official experimental branch bumps. They are not "upgrades" over the official version. And they might be bugged at time: only the official KCPP releases are to be considered correctly numbered, reliable and "fixed".
The LllamaCPP version + the additional PRs integrated follow my KCPP-Frankenstein versioning in the title, so everybody knows what version they deal with.

For KCPP official version, it's here : https://github.com/LostRuins/koboldcpp/releases

Features:

Enhanced benchmark (reflecting a maximum of indicators, including the KV cache option)

13 KV cache options (all should be considered experimental except F16 and KV Q8_0) 👍

  • F16 -> Fullproof (the usual KV quant since the beginning of LCPP/KCPP)
  • K Q8_0 with : V Q8_0 (stable, my current main, part of the LCPP/KCPP main triplet), Q5_1 (maybe unstable), Q5_0 (maybe unstable), Q4_1 (maybe stable), the rest is untested beyond benches), Q4_0
  • K Q5_1 with : V Q5_1, Q5_0, Q4_1, Q4_0
  • K Q5_0 with : V Q4_0
  • K Q4_1 with : V Q4_0
  • KV Q4_0 (quite stable, if we consider that it's part of the LCPP/KCPP main triplet)
  • Works in command line, normally also via the GUI, and normally saves on .KCPPS config files.

Arguments 👍

--quantkv",
help="Sets the KV cache data type quantization.

0=F16 (16 BPW),
8080=q8_0 (8.5 BPW),
8051=Kq8_0-Vq5_1 (7.25BPW),
8050=Kq8_0-Vq5_0 (7BPW),
8041=Kq8_0-Vq4_1 (6.75BPW),
8040=Kq8_0-Vq4_0 (6.5BPW),
5151=Kq5_1-Vq5_1 (6BPW),
5150=Kq5_1-Vq5_0 (5.75BPW),
5141=Kq5_1-Vq4_1 (5.5BPW),
5140=Kq5_1-Vq4_0 (5.25BPW),
5040=Kq5_0-Vq4_0 (5BPW),
4140=Kq4_1-Vq4_0 (4.75BPW),
4040=Q4_0 (4.5BPW).

choices=[0,8080,8051,8050,8041,8040,5151,5150,5141,5140,5040,4140,4040], default=0)`

Lowvram option's speed is (logically) boosted due to the smaller KV context in RAM. From 25%+ in Q8_0 to 50%+ in Q4_0.

Remarks :
You MUST use Flash attention
(tag : --flashattention in CL, or in the GUI)

Credits :
Of course, all credits go to Concedo/LostRuins and the other contributors to KoboldCPP, and to GGermanov and all the other contributors to LlamaCPP. Special big-up to Johannes Gaessler for the quantized KV cache!
I'm just poking, merging, and building around their work.

Builds:
All builds, aka. Cublas 12.1/12.2 (recommended for Ada Lovelace and Ampere for additional PP speed), Cublas 11.4.4/11.7 (more pertinent than the 12.2 for Pascal, Maxwell.. And 11.4.4 for Kepler), and the standard one are including OpenBLAS, CLBLAST, and Vulkan support provided by the devs.

Full Changelog: v1.67c_b3066...v1.67d_b3075

Assets 3
Loading