Q6_K AVX improvements

[netrunnereve]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

I've updated the Q6_K AVX implementation for better prompt processing and text generation speed. Most of the improvements came from using the bsums data to separately calculate the Q6 offsets (it's precomputed for a reason!), reducing the number of instructions.

The same techniques should also be applicable to the AVX2 implementation if someone wants to work on that. It looks like the ARM implementation already uses bsums correctly.

By the way I'm kinda curious if anyone here actually uses llama.cpp on an AVX only machine, or if I'm the only one 😞.

Results (Xeon E5 v2):

Master

q6_K
  vec_dot_q
    4096 values (0.02 MB)
      min cycles/32 vals   :     20.29
      avg cycles/32 vals   :     20.36
      float32 throughput   :     15.26 GB/s
      quantized throughput :      3.13 GB/s

PR

q6_K
  vec_dot_q
    4096 values (0.02 MB)
      min cycles/32 vals   :     16.43
      avg cycles/32 vals   :     16.51
      float32 throughput   :     19.07 GB/s
      quantized throughput :      3.91 GB/s

model	size	params	backend	threads	test	t/s	speedup
llama 7B Q6_K (Master)	5.53 GiB	7.24 B	CPU	8	pp512	7.59 ± 0.02
llama 7B Q6_K (Master)	5.53 GiB	7.24 B	CPU	8	tg128	5.75 ± 0.01
llama 7B Q6_K (PR)	5.53 GiB	7.24 B	CPU	8	pp512	10.29 ± 0.03	36%
llama 7B Q6_K (PR)	5.53 GiB	7.24 B	CPU	8	tg128	6.50 ± 0.02	13%

test-backend-ops and test-quantize-fns is passing.

[slaren]

I get these results with test-backend-ops perf:

master:

  MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  32706 runs -   305.94 us/run - 117.44 MFLOP/run - 383.87 GFLOPS
  MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                   87 runs - 115503.36 us/run -  60.13 GFLOP/run - 520.59 GFLOPS

PR:

  MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  37812 runs -   264.88 us/run - 117.44 MFLOP/run - 443.38 GFLOPS
  MUL_MAT(type_a=q6_K,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3]):                  107 runs - 94010.70 us/run -  60.13 GFLOP/run - 639.60 GFLOPS

[Nexesenex]

The same techniques should also be applicable to the AVX2 implementation if someone wants to work on that. It looks like the ARM implementation already uses bsums correctly.

Considering that most >3bpw quants use an output weight quantized in Q6_K (the Q6_K FTYPE using a Q6_K token embed weight as well), and that not offloading on GPU the output weight is done sometimes to save a bit of VRAM, would such AVX2 implementation reduce the loss of performances incurred when not offloading said output weight on GPU?

I'm using a Ryzen 5700G, which is AVX2 I believe.

[netrunnereve]

Considering that most >3bpw quants use an output weight quantized in Q6_K (the Q6_K FTYPE using a Q6_K token embed weight as well), and that not offloading on GPU the output weight is done sometimes to save a bit of VRAM, would such AVX2 implementation reduce the loss of performances incurred when not offloading said output weight on GPU?

I mean it should help in theory, but considering how the output weight is only a small part of the full model you probably won't notice a thing. I guess the easiest way to test this is to benchmark a Q4 or Q5 model with this PR.

[netrunnereve]

I had to update the CI target macOS-latest-cmake-x64 to run on macos-13 as macos-12 seems to have been depreciated by Github. The macos-13 machines are still x86 so this won't generate arm binaries (see #7940).

[SmallAndSoft]

By the way I'm kinda curious if anyone here actually uses llama.cpp on an AVX only machine, or if I'm the only one 😞.

I do. AMD FX is AVX only.