Why not have a build for "old" archs like AVX and SSE4?

[mlgitter]

I think builds supporting SSE4.2 and AVX1 CPUs are still useful.

Systems with up to 512Gb|1,5Tb RAM and 24|48|96 virtual cores, supporting 4090+ are too expensive and early to be thrown away.
BTW, rumors are, SSE4 code sometimes even faster, than AVX* code or at least, on a par.
If you have GPU(s), most of the computations are on GPUs.
Why don't provide universal build or a specific build for "old" hardware.

May be there is a fork that builds this way, but I can't find it.

Kobold seems for me, newcomer, a great tool but it crash at one of my SSE4 system.

I don't have a good skillset right away to build under Windows as I didn't touch C++ for twenty years.

Such a build would be still handy, I believe.

I see other people looking for the thing, too.
#631 (comment)
#358

[LostRuins]

We do have a build. You can download it here: https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp_oldcpu.exe

It comes with support for all the non-avx2 libraries.

[mlgitter]

Thanks!

It's finally started at my e5645x2 host with

--noavx2 --noblas --nommap --model=mistral-7b-instruct-v0.1.Q3_K_L.gguf

but seem doesn't respect and use SSE4.2

Using automatic RoPE scaling for GGUF. If the model has custom RoPE settings, they'll be used directly instead!

It means that the RoPE values written above will be replaced by the RoPE values indicated after loading.

System Info: AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | FMA = 0 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C
= 0 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from mistral-7b-instruct-v0.1.
Q3_K_L.gguf (version GGUF V2)

When i tried to put --usecublas it comes incompatible with --noblas but even with --noblas been excluded it crashed.

Using --useclblast 0 0 or other combinations like 0 1, 1 0 for my single 10708Gb didn't help too.

• What is the proper way to run this .exe or some other with turned on GPU support (which was the thing i've tried to ask about)?

[LostRuins]

So the oldcpu build with cublas doesn't work for you? Maybe your cpu has no avx1 support?

[LostRuins]

But you previously said you did have AVX1 support

I think builds supporting SSE4.2 and AVX1 CPUs are still useful.

[mlgitter]

But you previously said you did have AVX1 support

I think builds supporting SSE4.2 and AVX1 CPUs are still useful.

This doesn't mean I have AVX1 support on the target machine.

This means to fall back in execution to SSE4.2 or AVX1, whichever is present, if CPU doesn't support AVX2, thats all.

(For most purposes "old" SSE4.2 and AVX1 sometimes comes somewhat on par AVX2. but it not essential).

I believe this is somewhat a matter of using -msse4.2 compile option for SSE4.2.
May be some dependent libs will need to be compiled with -msse4.2 option.

After all, it will make Kobold even more popular as accessible to a broader public, I think.

[LostRuins]

Ah I think I get the confusion. For llamacpp and KoboldCpp, the supported intrinsic must be set at build time and they control what code gets compiled and included. So you'd have to make a new build for each target, which is what is currently done.

There's a failsafe mode with no intrinsics at all too.

[mlgitter]

If you call intrinsic a max SIMD variation available, I think it's desireable to include SSE4.1|2, since current fallback (to noSIMD, just X64|86, I suppose) old_cpu has no GPU support compiled without GPU options and is inappropriately slow, so I vote for a build with SSE4.1|2 and GPU support, which is what other users already asked for here and there in your discussions.
And many people I believe see just a slow version or exceptions and simply use, vote and support other tools.

[mlgitter]

I think your tool Kobold which I use on another machine with avx is good and just deserve more cpu options to support.

[LostRuins]

I'm wondering though, how many people will have a GPU new enough to support modern CUDA and x86-64 windows, but also have a CPU old enough that it does not support AVX1 (which came out around Sandy Bridge circa 2012)?

For the oldcpu binary build - I'm willing to drop avx1 completely for oldcpu builds (compile cuda with no simd) but not willing to add yet another new build target to the releases (which is around 3GB of files already!) Ideally this should be justified by real world use scenarios to please the largest groups of users. At least it sounds like a very uncommon setup

@henk717 might be able to advise too

[mlgitter]

I'm wondering though, how many people will have a GPU new enough to support modern CUDA and x86-64 windows, but also have a CPU old enough that it does not support AVX1 (which came out around Sandy Bridge circa 2012)?

For the oldcpu binary build - I'm willing to drop avx1 completely for oldcpu builds (compile cuda with no simd) but not willing to add yet another new build target to the releases (which is around 3GB of files already!) Ideally this should be justified by real world use scenarios to please the largest groups of users. At least it sounds like a very uncommon setup

@henk717 might be able to advise too

There are still many dual (or even sometimes quad) cpu systems around, which are behind mainstream setups say, for 2-3 cpu generations in the sense of having mainstream SIMD, but are still faster, as cpu speed increase is 10-20% per generation. So midrange or top dual SSE4.1|2 cpus are somewhat similar low-end AVX-512 single cpu systems, still able to drive whichever top gpus (yes, proper cases and power supplies, etc).

If I would do a program, I would provide steps in fallback SIMD's stairs, whichever available.
Nowadays all OSes provide support for a full range of cpu generations, and for people would be quite natural to expect a software to work according features they paid for and have in their hardware. SIMDs aren't the same as asm86|64.
After all, people prefer to invest time and effort in a caret that still can move, may be a bit slower over time but won't become a pumpkin one day.

The developers always have their own ideas about everything they do - well, I just share a bit of my own view of the subject.
I just came across KoboldCpp a couple of weeks ago and now have a feeling I write too much here and not ready for a discussion.

[henk717]

I don't think changing what we have is a good idea since its already optimal. 99.9% of users have at minimum AVX1 since those machines are over 10 years old now the exception mostly being me running it on my retro PC frrom 2005 every now and then. Those systems almost never have a GPU new enough to be able to run the software and are already supported by the fallback option.

So given how minimal this release target's userbase is I don't think its a good idea to disrupt the people relying on the old CPU builds with even slower CPU offloading. It would make more sense for these users to either self compile or for us to do an occasional one off build like we used to do for AVX1 before it became popular enough to warrant a regular build in the release cycle.

What we could do however if we have such a build is benchmark the CPU only mode for SSE4.1 vs AVX1. If its a small difference it would make sense to swap.