Building koboldcpp in Termux on Android

[inspir3dArt]

Hi,

I played around with building koboldcpp in Termux and got improvements in prompt generation performance.

I got (open?)blas to work for prompt processing. It says [Blas] during prompt processing and process by default up to 512 tokens at once, but I can't really say if it's faster.

The prompt generation feels a lot faster and more smooth, I can't tell why or what optimisations it is using.

Have tested it on a Samsung Galaxy S24 Ultra and a OnePlus 10 pro, with a few current Mistral and Llama based 7b / 8b models.

For older or non high end devices it's important to check if they support ARMV9 and replace it with the arm version your device supports if not in the build command.

I wonder if it can be further optimized by building it against the android ndk or the chip vendors build tools (in my case snapdragon), but I have no clue how to even start to play around with it.

This is how I currently build koboldcpp in Termux:

Change repo (choose the Mirror by BFSU):
termux-change-repo

Update packages:
pkg up

Install dependencies:
pkg install wget git python openssl clang opencl-headers ocl-icd clinfo blas-openblas clblast libopenblas libopenblas-static

Clone the koboldcpp repo:
git clone https://github.com/LostRuins/koboldcpp.git

Navigate to the koboldcpp folder:
cd koboldcpp

Build koboldcpp:
make TARGET=ARMV9 LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1 LLAMA_VULKAN=1

[gustrd]

Hello @inspir3dArt!

I'm the author of some old instructions for building llama.cpp / koboldcpp on Android using CLBlast.

Even though I still use it myself, the instructions were removed because the OpenCL/Vulkan implementations on Android are still known to be buggy.

For example, I can run a model from StableLM with CLBlast with no layer offload and no k-quant. But if I try with layer offload, it produces gibberish. Even stranger, with k-quant it can respond to the first instruction, but the subsequent ones also become gibberish.

Several families of models don't work at all with CLBlast. So these instructions were causing trouble.

I'm also using an AMRv9 device (Zenfone 9).

Anyway, you can check the old conversations about this in this thread from upstream: ggerganov#7016

[JustANoobber]

I know you probably won't see this but how exactly should we load up kobold once we are finished with the process?

[LostRuins]

Just run
python koboldcpp.py --model /path/to/your/file.gguf

[JustANoobber]

Thank you so much^^ a bit off topic but how exactly can I set up persona? It's not giving me the option

[LostRuins]

After you import your Tavern character card (you can just drag and drop it into the GUI or click Save/Load), you will be able to edit in in the Context memory. (click the context button)