Skip to content

Kobold.CPP_FrankenFork_v1.68k_b3184+1
Compare
Choose a tag to compare
@Nexesenex Nexesenex released this 19 Jun 17:26
· 94 commits to frankenrope since this release
v1.68k_b3184+1
9524e1a

Frankenstein 1.68k "Fork" of KoboldCPP Experimental up to the 19/06/2024, 16h GMT+2.
Based on Llama.CPP b3184.

DISCLAIMER:

The KoboldCPP-Frankenstein builds are not supported by the KoboldCPP team, Github, or discord channel. They are for greedy-test and amusement only.
My KCPP-Frankenstein version number bumps as soon as the version number in the official experimental branch bumps. They are not "upgrades" over the official version. And they might be bugged at time: only the official KCPP releases are to be considered correctly numbered, reliable and "fixed".
The LllamaCPP version + the additional PRs integrated follow my KCPP-Frankenstein versioning in the title, so everybody knows what version they deal with.

For KCPP official version, it's here : https://github.com/LostRuins/koboldcpp/releases

FRANKENSTEIN FEATURES:

  • Enhanced benchmark (reflecting a maximum of indicators, including the KV cache option. Now integrated in a slightly revamped version in the official version.

  • 18 KV cache options (all should be considered experimental except F16 and KV Q8_0) 👍

F16 -> Fullproof (the usual KV quant since the beginning of LCPP/KCPP)
K F16 with : V Q8_0, Q5_1, Q5_0, Q4_1, Q4_0
K Q8_0 with : V Q8_0 (stable, my current main, part of the LCPP/KCPP main triplet), Q5_1 (maybe unstable), Q5_0 (maybe unstable), Q4_1 (maybe stable), the rest is untested beyond benches), Q4_0
K Q5_1 with : V Q5_1, Q5_0, Q4_1, Q4_0
K Q5_0 with : V Q4_0
K Q4_1 with : V Q4_0
KV Q4_0 (quite stable, if we consider that it's part of the LCPP/KCPP main triplet)
Works in command line, normally also via the GUI, and normally saves on .KCPPS config files.

A better Autorope thanks to askmyteapot's PR on KoboldCPP official github, and, for Llama models, with an additional negative offset to lower a bit the rope, as well as a positive offset for SOLAR models, and improve the perplexity (L1,L2) or avoid to degrade too much the reasoning abilities (L3) at equal context.

ARGUMENTS 👍

Note : I had to use a simple 0-17 numbering scheme to allow the GUI and the kcpps preset saving to work properly. The problems with the previous 4 numbers quant scheme are fixed.

--quantkv",
help="Sets the KV cache data type quantization.

0 = 1616/F16 (16 BPW),

1 = 1680/Kf16-Vq8_0 (12.25BPW),
2 = 1651/Kf16-Vq5_1 (11BPW),
3 = 1650/Kf16-Vq5_0 (10.75BPW),
4 = 1641/Kf16-Vq4_1 (10.5BPW),
5 = 1640/Kf16-Vq4_0 (10.25BPW),

6 = 8080/KVq8_0 (8.5 BPW),
7 = 8051/Kq8_0-Vq5_1 (7.25BPW),
8 = 8050/Kq8_0-Vq5_0 (7BPW),
9 = 8041/Kq8_0-Vq4_1 (6.75BPW),
10 = 8040/Kq8_0-Vq4_0 (6.5BPW),

11 = 5151/KVq5_1 (6BPW),
12 = 5150/Kq5_1-Vq5_0 (5.75BPW),
13 = 5141/Kq5_1-Vq4_1 (5.5BPW),
14 = 5140/Kq5_1-Vq4_0 (5.25BPW),

15 = 5040/Kq5_0-Vq4_0 (5BPW),

16 = 4140/Kq4_1-Vq4_0 (4.75BPW),

17 = 4040/KVq4_0 (4.5BPW)

choices=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17], default=0)

Lowvram option's speed is (logically) boosted due to the smaller KV context in RAM. From 25%+ in KV Q8_0 to 50%+ in KV Q4_0.

REMARKS :
You MUST use Flash attention
(tag : --flashattention in CL, or in the GUI)
Contextshift doesn't work with anything else than KV F16, but Smartcontext does.

CREDITS :
Of course, all credits go to Concedo/LostRuins and the other contributors to KoboldCPP, and to GGermanov and all the other contributors to LlamaCPP. Special big-up to Johannes Gaessler for the quantized KV cache!
I'm just poking, merging, and building around their work.

BUILDS :
All builds, aka. Cublas 12.1/12.2 (recommended for Ada Lovelace and Ampere for additional PP speed), Cublas 11.4.4/11.7 (more pertinent than the 12.2 for Pascal, Maxwell.. And 11.4.4 for Kepler), and the standard one are including OpenBLAS, CLBLAST, and Vulkan support provided by the devs.

What's Changed beyond b_3184

AVX IQ Quants PR 7845 by netrunnereve

Full Changelog: v1.68g_b3158+4...v1.68k_b3184+1

Assets 4
Loading