BUG : ggml_cuda_init: failed to initialize CUDA: forward compatibility was attempted on non supported HW

[sprappcom]

export as compute_89 and make compile with CUDA, everything works but when i did inference with llama-cli, i get

ggml_cuda_init: failed to initialize CUDA: forward compatibility was attempted on non supported HW

so it's not using gpu ram etc (i've checked, only pure cpu)

what does this mean and how to remedy?
ubuntu 24.04 nvcc 12.5, latest llama.cpp, laptop amd ryzen 7000 series with 4060 gpu

[dspasyuk]

@sprappcom try building for all arch, so:

git clone https://github.com/ggerganov/llama.cpp.git; cd llama.cpp; sed -i 's/-arch=native/-arch=all/g' Makefile; make clean && LLAMA_CUDA=1 make -j 6

[sprappcom]

@dspasyuk did it, still the same issue:
ggml_cuda_host_malloc: failed to allocate 4096.00 MiB of pinned memory: forward compatibility was attempted on non supported HW

can u check why? i cant offload with ngl

[dspasyuk]

that is strange what does ldd llama-cli says?

[sprappcom]

same problem ggml_cuda_host_malloc: failed to allocate 4096.00 MiB of pinned memory: forward compatibility was attempted on non supported HW

then it uses cpu to do inference

[dspasyuk]

@sprappcom Unable to allocate error is most likely due to low available memory compared to the model size, so if you have 10 Gb of VRAM you should be able to load 7B parameter model. Anything less and you need to reduce -ngl parameter or lower quantization etc:

Try this:

../llama.cpp/llama-cli --model ./models/meta-llama-3-8b-instruct-q5_k_s.gguf --n-gpu-layers 10 -cnv -b 2048 --ctx_size 0 --temp 0 --top_k 10 --multiline-input --chat-template llama3 -p 'You are Alice, a large language model.'

[sprappcom]

@dspasyuk
./llama-cli -m ../llamamodels/mistral-7b-openorca.Q4_0.gguf -ngl 1

used to work before without ./llama-cli. i mean used to work before when everything was still ./main
i have 8gb 4060 rtx laptop. this 7b can fit complete previously with ./main.

i did this and still...
./llama-cli -m ../llamamodels/mistral-7b-openorca.Q4_0.gguf -ngl 1

llm_load_tensors: offloading 1 repeating layers to GPU
llm_load_tensors: offloaded 1/33 layers to GPU
llm_load_tensors:        CPU buffer size =  3917.88 MiB
..................................................................................................
llama_new_context_with_model: n_ctx      = 32768
llama_new_context_with_model: n_batch    = 2048
llama_new_context_with_model: n_ubatch   = 512
llama_new_context_with_model: flash_attn = 0
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_cuda_host_malloc: failed to allocate 4096.00 MiB of pinned memory: forward compatibility was attempted on non supported HW
llama_kv_cache_init:        CPU KV buffer size =  4096.00 MiB
llama_new_context_with_model: KV self size  = 4096.00 MiB, K (f16): 2048.00 MiB, V (f16): 2048.00 MiB
ggml_cuda_host_malloc: failed to allocate 0.12 MiB of pinned memory: forward compatibility was attempted on non supported HW
llama_new_context_with_model:        CPU  output buffer size =     0.12 MiB
ggml_cuda_host_malloc: failed to allocate 2144.01 MiB of pinned memory: forward compatibility was attempted on non supported HW
llama_new_context_with_model:  CUDA_Host compute buffer size =  2144.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 1

system_info: n_threads = 6 / 12 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | 
sampling: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampling order: 
CFG -> Penalties -> top_k -> tfs_z -> typical_p -> top_p -> min_p -> temperature 
generate: n_ctx = 32768, n_batch = 2048, n_predict = -1, n_keep = 1


 This week, we've been getting lots of questions from our readers about their pets. Here are some answers to some of the most common questions we've received.
<dummy32000> [end of text]

llama_print_timings:        load time =    7628.46 ms
llama_print_timings:      sample time =       0.81 ms /    36 runs   (    0.02 ms per token, 44554.46 tokens per second)
llama_print_timings: prompt eval time =       0.00 ms /     0 tokens (    -nan ms per token,     -nan tokens per second)
llama_print_timings:        eval time =    3005.83 ms /    36 runs   (   83.50 ms per token,    11.98 tokens per second)
llama_print_timings:       total time =    3007.95 ms /    36 tokens
Log end

[dspasyuk]

could you type "ldd ./llama-cli" and nvidia-smi in the terminal? It looks to me that either your drive version is not compatible with Cuda or Cuda not compatible with the driver. You might need to update your driver or Cuda. My driver is 545.23.08 and my cuda is 12.2

[Oussamayousre]

@sprappcom Please did you end up resolving the problem , i have the exact same problem and I almost tried everything , nothing seems to work !