LlamaCpp on Mac generates nonsense

[maximegmd]

Describe the issue as clearly as possible:

When using standard text generation based on a LlamaCpp model, the generation is completely random and makes no sense.

Steps/code to reproduce the bug:

import outlines

model = outlines.models.llamacpp("TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q4_K_M.gguf", device="cpu")
generator = outlines.generate.text(model)
sequence = generator("Q: Name the planets in the solar system? A: ", max_tokens=512)
print(sequence)

Expected result:

sequence doesn't look like this:


Mercury Retro 12: What'sPyEvening.io/Lens Finallyslycrypt: https://Comm
Monkeylog= https://www.myvergeoc-go/
Coinc described here is a Minecraft_Git/

  
You are you already a hacking.gg/Waltbits?

Hit me being rocket    Topic-Petalotanie/subscribe
A/ Telegram.toyosmc
Answer: Logout

### Error message:

```shell
llama_model_loader: loaded meta data with 20 key-value pairs and 291 tensors from /Users/maxime/.cache/lm-studio/models/TheBloke/Mistral-7B-Instruct-v0.1-GGUF/mistral-7b-instruct-v0.1.Q4_K_M.gguf (version GGUF V2)
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = mistralai_mistral-7b-instruct-v0.1
llama_model_loader: - kv   2:                       llama.context_length u32              = 32768
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 4096
llama_model_loader: - kv   4:                          llama.block_count u32              = 32
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 14336
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 32
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 15
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32000]   = ["<unk>", "<s>", "</s>", "<0x00>", "<...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32000]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 0
llama_model_loader: - kv  19:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:   65 tensors
llama_model_loader: - type q4_K:  193 tensors
llama_model_loader: - type q6_K:   33 tensors
llm_load_vocab: special tokens definition check successful ( 259/32000 ).
llm_load_print_meta: format           = GGUF V2
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 32000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 4096
llm_load_print_meta: n_head           = 32
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 32
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 4
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 14336
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 10000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 7B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 7.24 B
llm_load_print_meta: model size       = 4.07 GiB (4.83 BPW) 
llm_load_print_meta: general.name     = mistralai_mistral-7b-instruct-v0.1
llm_load_print_meta: BOS token        = 1 '<s>'
llm_load_print_meta: EOS token        = 2 '</s>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: LF token         = 13 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.22 MiB
ggml_backend_metal_buffer_from_ptr: allocated buffer, size =  4095.08 MiB, ( 4095.16 / 49152.00)
llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    70.31 MiB
llm_load_tensors:      Metal buffer size =  4095.07 MiB
...............................................................................................
llama_new_context_with_model: n_ctx      = 512
llama_new_context_with_model: freq_base  = 10000.0
llama_new_context_with_model: freq_scale = 1
ggml_metal_init: allocating
ggml_metal_init: found device: Apple M3 Max
ggml_metal_init: picking default device: Apple M3 Max
ggml_metal_init: default.metallib not found, loading from source
ggml_metal_init: GGML_METAL_PATH_RESOURCES = nil
ggml_metal_init: loading '/Users/maxime/anaconda3/envs/torch/lib/python3.11/site-packages/llama_cpp/ggml-metal.metal'
ggml_metal_init: GPU name:   Apple M3 Max
ggml_metal_init: GPU family: MTLGPUFamilyApple9  (1009)
ggml_metal_init: GPU family: MTLGPUFamilyCommon3 (3003)
ggml_metal_init: GPU family: MTLGPUFamilyMetal3  (5001)
ggml_metal_init: simdgroup reduction support   = true
ggml_metal_init: simdgroup matrix mul. support = true
ggml_metal_init: hasUnifiedMemory              = true
ggml_metal_init: recommendedMaxWorkingSetSize  = 51539.61 MB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    64.00 MiB, ( 4165.03 / 49152.00)
llama_kv_cache_init:      Metal KV buffer size =    64.00 MiB
llama_new_context_with_model: KV self size  =   64.00 MiB, K (f16):   32.00 MiB, V (f16):   32.00 MiB
llama_new_context_with_model:        CPU input buffer size   =     9.01 MiB
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =     0.02 MiB, ( 4165.05 / 49152.00)
ggml_backend_metal_buffer_type_alloc_buffer: allocated buffer, size =    80.31 MiB, ( 4245.34 / 49152.00)
llama_new_context_with_model:      Metal compute buffer size =    80.30 MiB
llama_new_context_with_model:        CPU compute buffer size =     8.80 MiB
llama_new_context_with_model: graph splits (measure): 3
Failed to convert token (b'\x80'): 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Failed to convert token (b'\x81'): 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte
Failed to convert token (b'\x82'): 'utf-8' codec can't decode byte 0x82 in position 0: invalid start byte
Failed to convert token (b'\x83'): 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte
Failed to convert token (b'\x84'): 'utf-8' codec can't decode byte 0x84 in position 0: invalid start byte
Failed to convert token (b'\x85'): 'utf-8' codec can't decode byte 0x85 in position 0: invalid start byte
Failed to convert token (b'\x86'): 'utf-8' codec can't decode byte 0x86 in position 0: invalid start byte
Failed to convert token (b'\x87'): 'utf-8' codec can't decode byte 0x87 in position 0: invalid start byte
Failed to convert token (b'\x88'): 'utf-8' codec can't decode byte 0x88 in position 0: invalid start byte
Failed to convert token (b'\x89'): 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
Failed to convert token (b'\x8a'): 'utf-8' codec can't decode byte 0x8a in position 0: invalid start byte
Failed to convert token (b'\x8b'): 'utf-8' codec can't decode byte 0x8b in position 0: invalid start byte
Failed to convert token (b'\x8c'): 'utf-8' codec can't decode byte 0x8c in position 0: invalid start byte
Failed to convert token (b'\x8d'): 'utf-8' codec can't decode byte 0x8d in position 0: invalid start byte
Failed to convert token (b'\x8e'): 'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte
Failed to convert token (b'\x8f'): 'utf-8' codec can't decode byte 0x8f in position 0: invalid start byte
Failed to convert token (b'\x90'): 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
Failed to convert token (b'\x91'): 'utf-8' codec can't decode byte 0x91 in position 0: invalid start byte
Failed to convert token (b'\x92'): 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte
Failed to convert token (b'\x93'): 'utf-8' codec can't decode byte 0x93 in position 0: invalid start byte
Failed to convert token (b'\x94'): 'utf-8' codec can't decode byte 0x94 in position 0: invalid start byte
Failed to convert token (b'\x95'): 'utf-8' codec can't decode byte 0x95 in position 0: invalid start byte
Failed to convert token (b'\x96'): 'utf-8' codec can't decode byte 0x96 in position 0: invalid start byte
Failed to convert token (b'\x97'): 'utf-8' codec can't decode byte 0x97 in position 0: invalid start byte
Failed to convert token (b'\x98'): 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
Failed to convert token (b'\x99'): 'utf-8' codec can't decode byte 0x99 in position 0: invalid start byte
Failed to convert token (b'\x9a'): 'utf-8' codec can't decode byte 0x9a in position 0: invalid start byte
Failed to convert token (b'\x9b'): 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte
Failed to convert token (b'\x9c'): 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte
Failed to convert token (b'\x9d'): 'utf-8' codec can't decode byte 0x9d in position 0: invalid start byte
Failed to convert token (b'\x9e'): 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
Failed to convert token (b'\x9f'): 'utf-8' codec can't decode byte 0x9f in position 0: invalid start byte
Failed to convert token (b'\xa0'): 'utf-8' codec can't decode byte 0xa0 in position 0: invalid start byte
Failed to convert token (b'\xa1'): 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte
Failed to convert token (b'\xa2'): 'utf-8' codec can't decode byte 0xa2 in position 0: invalid start byte
Failed to convert token (b'\xa3'): 'utf-8' codec can't decode byte 0xa3 in position 0: invalid start byte
Failed to convert token (b'\xa4'): 'utf-8' codec can't decode byte 0xa4 in position 0: invalid start byte
Failed to convert token (b'\xa5'): 'utf-8' codec can't decode byte 0xa5 in position 0: invalid start byte
Failed to convert token (b'\xa6'): 'utf-8' codec can't decode byte 0xa6 in position 0: invalid start byte
Failed to convert token (b'\xa7'): 'utf-8' codec can't decode byte 0xa7 in position 0: invalid start byte
Failed to convert token (b'\xa8'): 'utf-8' codec can't decode byte 0xa8 in position 0: invalid start byte
Failed to convert token (b'\xa9'): 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte
Failed to convert token (b'\xaa'): 'utf-8' codec can't decode byte 0xaa in position 0: invalid start byte
Failed to convert token (b'\xab'): 'utf-8' codec can't decode byte 0xab in position 0: invalid start byte
Failed to convert token (b'\xac'): 'utf-8' codec can't decode byte 0xac in position 0: invalid start byte
Failed to convert token (b'\xad'): 'utf-8' codec can't decode byte 0xad in position 0: invalid start byte
Failed to convert token (b'\xae'): 'utf-8' codec can't decode byte 0xae in position 0: invalid start byte
Failed to convert token (b'\xaf'): 'utf-8' codec can't decode byte 0xaf in position 0: invalid start byte
Failed to convert token (b'\xb0'): 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
Failed to convert token (b'\xb1'): 'utf-8' codec can't decode byte 0xb1 in position 0: invalid start byte
Failed to convert token (b'\xb2'): 'utf-8' codec can't decode byte 0xb2 in position 0: invalid start byte
Failed to convert token (b'\xb3'): 'utf-8' codec can't decode byte 0xb3 in position 0: invalid start byte
Failed to convert token (b'\xb4'): 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
Failed to convert token (b'\xb5'): 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte
Failed to convert token (b'\xb6'): 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte
Failed to convert token (b'\xb7'): 'utf-8' codec can't decode byte 0xb7 in position 0: invalid start byte
Failed to convert token (b'\xb8'): 'utf-8' codec can't decode byte 0xb8 in position 0: invalid start byte
Failed to convert token (b'\xb9'): 'utf-8' codec can't decode byte 0xb9 in position 0: invalid start byte
Failed to convert token (b'\xba'): 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte
Failed to convert token (b'\xbb'): 'utf-8' codec can't decode byte 0xbb in position 0: invalid start byte
Failed to convert token (b'\xbc'): 'utf-8' codec can't decode byte 0xbc in position 0: invalid start byte
Failed to convert token (b'\xbd'): 'utf-8' codec can't decode byte 0xbd in position 0: invalid start byte
Failed to convert token (b'\xbe'): 'utf-8' codec can't decode byte 0xbe in position 0: invalid start byte
Failed to convert token (b'\xbf'): 'utf-8' codec can't decode byte 0xbf in position 0: invalid start byte
Failed to convert token (b'\xc0'): 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
Failed to convert token (b'\xc1'): 'utf-8' codec can't decode byte 0xc1 in position 0: invalid start byte
Failed to convert token (b'\xc2'): 'utf-8' codec can't decode byte 0xc2 in position 0: unexpected end of data
Failed to convert token (b'\xc3'): 'utf-8' codec can't decode byte 0xc3 in position 0: unexpected end of data
Failed to convert token (b'\xc4'): 'utf-8' codec can't decode byte 0xc4 in position 0: unexpected end of data
Failed to convert token (b'\xc5'): 'utf-8' codec can't decode byte 0xc5 in position 0: unexpected end of data
Failed to convert token (b'\xc6'): 'utf-8' codec can't decode byte 0xc6 in position 0: unexpected end of data
Failed to convert token (b'\xc7'): 'utf-8' codec can't decode byte 0xc7 in position 0: unexpected end of data
Failed to convert token (b'\xc8'): 'utf-8' codec can't decode byte 0xc8 in position 0: unexpected end of data
Failed to convert token (b'\xc9'): 'utf-8' codec can't decode byte 0xc9 in position 0: unexpected end of data
Failed to convert token (b'\xca'): 'utf-8' codec can't decode byte 0xca in position 0: unexpected end of data
Failed to convert token (b'\xcb'): 'utf-8' codec can't decode byte 0xcb in position 0: unexpected end of data
Failed to convert token (b'\xcc'): 'utf-8' codec can't decode byte 0xcc in position 0: unexpected end of data
Failed to convert token (b'\xcd'): 'utf-8' codec can't decode byte 0xcd in position 0: unexpected end of data
Failed to convert token (b'\xce'): 'utf-8' codec can't decode byte 0xce in position 0: unexpected end of data
Failed to convert token (b'\xcf'): 'utf-8' codec can't decode byte 0xcf in position 0: unexpected end of data
Failed to convert token (b'\xd0'): 'utf-8' codec can't decode byte 0xd0 in position 0: unexpected end of data
Failed to convert token (b'\xd1'): 'utf-8' codec can't decode byte 0xd1 in position 0: unexpected end of data
Failed to convert token (b'\xd2'): 'utf-8' codec can't decode byte 0xd2 in position 0: unexpected end of data
Failed to convert token (b'\xd3'): 'utf-8' codec can't decode byte 0xd3 in position 0: unexpected end of data
Failed to convert token (b'\xd4'): 'utf-8' codec can't decode byte 0xd4 in position 0: unexpected end of data
Failed to convert token (b'\xd5'): 'utf-8' codec can't decode byte 0xd5 in position 0: unexpected end of data
Failed to convert token (b'\xd6'): 'utf-8' codec can't decode byte 0xd6 in position 0: unexpected end of data
Failed to convert token (b'\xd7'): 'utf-8' codec can't decode byte 0xd7 in position 0: unexpected end of data
Failed to convert token (b'\xd8'): 'utf-8' codec can't decode byte 0xd8 in position 0: unexpected end of data
Failed to convert token (b'\xd9'): 'utf-8' codec can't decode byte 0xd9 in position 0: unexpected end of data
Failed to convert token (b'\xda'): 'utf-8' codec can't decode byte 0xda in position 0: unexpected end of data
Failed to convert token (b'\xdb'): 'utf-8' codec can't decode byte 0xdb in position 0: unexpected end of data
Failed to convert token (b'\xdc'): 'utf-8' codec can't decode byte 0xdc in position 0: unexpected end of data
Failed to convert token (b'\xdd'): 'utf-8' codec can't decode byte 0xdd in position 0: unexpected end of data
Failed to convert token (b'\xde'): 'utf-8' codec can't decode byte 0xde in position 0: unexpected end of data
Failed to convert token (b'\xdf'): 'utf-8' codec can't decode byte 0xdf in position 0: unexpected end of data
Failed to convert token (b'\xe0'): 'utf-8' codec can't decode byte 0xe0 in position 0: unexpected end of data
Failed to convert token (b'\xe1'): 'utf-8' codec can't decode byte 0xe1 in position 0: unexpected end of data
Failed to convert token (b'\xe2'): 'utf-8' codec can't decode byte 0xe2 in position 0: unexpected end of data
Failed to convert token (b'\xe3'): 'utf-8' codec can't decode byte 0xe3 in position 0: unexpected end of data
Failed to convert token (b'\xe4'): 'utf-8' codec can't decode byte 0xe4 in position 0: unexpected end of data
Failed to convert token (b'\xe5'): 'utf-8' codec can't decode byte 0xe5 in position 0: unexpected end of data
Failed to convert token (b'\xe6'): 'utf-8' codec can't decode byte 0xe6 in position 0: unexpected end of data
Failed to convert token (b'\xe7'): 'utf-8' codec can't decode byte 0xe7 in position 0: unexpected end of data
Failed to convert token (b'\xe8'): 'utf-8' codec can't decode byte 0xe8 in position 0: unexpected end of data
Failed to convert token (b'\xe9'): 'utf-8' codec can't decode byte 0xe9 in position 0: unexpected end of data
Failed to convert token (b'\xea'): 'utf-8' codec can't decode byte 0xea in position 0: unexpected end of data
Failed to convert token (b'\xeb'): 'utf-8' codec can't decode byte 0xeb in position 0: unexpected end of data
Failed to convert token (b'\xec'): 'utf-8' codec can't decode byte 0xec in position 0: unexpected end of data
Failed to convert token (b'\xed'): 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data
Failed to convert token (b'\xee'): 'utf-8' codec can't decode byte 0xee in position 0: unexpected end of data
Failed to convert token (b'\xef'): 'utf-8' codec can't decode byte 0xef in position 0: unexpected end of data
Failed to convert token (b'\xf0'): 'utf-8' codec can't decode byte 0xf0 in position 0: unexpected end of data
Failed to convert token (b'\xf1'): 'utf-8' codec can't decode byte 0xf1 in position 0: unexpected end of data
Failed to convert token (b'\xf2'): 'utf-8' codec can't decode byte 0xf2 in position 0: unexpected end of data
Failed to convert token (b'\xf3'): 'utf-8' codec can't decode byte 0xf3 in position 0: unexpected end of data
Failed to convert token (b'\xf4'): 'utf-8' codec can't decode byte 0xf4 in position 0: unexpected end of data
Failed to convert token (b'\xf5'): 'utf-8' codec can't decode byte 0xf5 in position 0: invalid start byte
Failed to convert token (b'\xf6'): 'utf-8' codec can't decode byte 0xf6 in position 0: invalid start byte
Failed to convert token (b'\xf7'): 'utf-8' codec can't decode byte 0xf7 in position 0: invalid start byte
Failed to convert token (b'\xf8'): 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
Failed to convert token (b'\xf9'): 'utf-8' codec can't decode byte 0xf9 in position 0: invalid start byte
Failed to convert token (b'\xfa'): 'utf-8' codec can't decode byte 0xfa in position 0: invalid start byte
Failed to convert token (b'\xfb'): 'utf-8' codec can't decode byte 0xfb in position 0: invalid start byte
Failed to convert token (b'\xfc'): 'utf-8' codec can't decode byte 0xfc in position 0: invalid start byte
Failed to convert token (b'\xfd'): 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Failed to convert token (b'\xfe'): 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte
Failed to convert token (b'\xff'): 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

Outlines/Python version information:

Version information

``` 0.0.28 Python 3.11.7 (main, Dec 15 2023, 12:09:56) [Clang 14.0.6 ] absl-py==2.1.0 accelerate==0.26.1 addict==2.4.0 aiobotocore==2.7.0 aiofiles==23.2.1 aiohttp==3.9.3 aioitertools==0.11.0 aiosignal==1.3.1 alembic==1.13.1 altair==5.2.0 annotated-types==0.6.0 anyio==4.2.0 appdirs==1.4.4 appnope @ file:///home/conda/feedstock_root/build_artifacts/appnope_1707233003401/work art==6.1 asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1698341106958/work attrs==23.2.0 -e git+https://github.com/maximegmd/axolotl.git@e25fe924718f97fd426bd40b434f7f3a303ed98d#egg=axolotl bitsandbytes==0.42.0 black==24.1.1 blinker==1.7.0 botocore==1.31.64 cachetools==5.3.2 certifi==2024.2.2 cfgv==3.4.0 charset-normalizer==3.3.2 click==8.1.7 cloudpickle==3.0.0 colorama==0.4.6 coloredlogs==15.0.1 comm @ file:///home/conda/feedstock_root/build_artifacts/comm_1704278392174/work contourpy==1.2.0 cycler==0.12.1 databricks-cli==0.18.0 datasets==2.16.1 debugpy @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_563_nwtkoc/croot/debugpy_1690905063850/work decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work dill==0.3.7 diskcache==5.6.3 distlib==0.3.8 distro==1.9.0 docker==7.0.0 docker-pycreds==0.4.0 docstring-parser==0.15 einops==0.7.0 entrypoints==0.4 evaluate==0.4.0 exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1704921103267/work executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1698579936712/work fastapi==0.109.1 ffmpy==0.3.1 filelock==3.13.1 fire==0.5.0 Flask==3.0.2 fonttools==4.47.2 frozenlist==1.4.1 fschat==0.2.34 fsspec==2023.10.0 gcsfs==2023.10.0 gitdb==4.0.11 GitPython==3.1.41 google-api-core==2.16.2 google-auth==2.27.0 google-auth-oauthlib==1.2.0 google-cloud-core==2.4.1 google-cloud-storage==2.14.0 google-crc32c==1.5.0 google-resumable-media==2.7.0 googleapis-common-protos==1.62.0 gradio==3.50.2 gradio_client==0.6.1 grpcio==1.60.1 gunicorn==21.2.0 h11==0.14.0 hf_transfer==0.1.5 httpcore==1.0.2 httpx==0.26.0 huggingface-hub==0.20.3 humanfriendly==10.0 identify==2.5.33 idna==3.6 importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1703269254275/work importlib-resources==6.1.1 interegular==0.3.3 ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1707326353119/work ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1706795662110/work itsdangerous==2.1.2 jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work Jinja2==3.1.3 jmespath==1.0.1 joblib==1.3.2 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1699283905679/work jupyter_core @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_782yoyc_98/croot/jupyter_core_1698937318631/work kiwisolver==1.4.5 lark==1.1.9 llama_cpp_python==0.2.39 llvmlite==0.42.0 Mako==1.3.2 Markdown==3.5.2 markdown-it-py==3.0.0 markdown2==2.4.12 MarkupSafe==2.1.5 matplotlib==3.8.2 matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work mdurl==0.1.2 mlflow==2.10.0 mpmath==1.3.0 multidict==6.0.5 multiprocess==0.70.15 mypy==1.8.0 mypy-extensions==1.0.0 nest_asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1705850609492/work networkx==3.2.1 nh3==0.2.15 nodeenv==1.8.0 numba==0.59.0 numpy==1.26.3 oauthlib==3.2.2 openai==1.12.0 optimum==1.16.2 orjson==3.9.13 outlines==0.0.28 packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1696202382185/work pandas==2.2.0 parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work pathspec==0.12.1 peft @ git+https://github.com/huggingface/peft.git@ce925d844a0bc54b951fcb69229dfe740c9afa45 pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1706113125309/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work pillow==10.2.0 platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1706713388748/work pre-commit==3.6.0 prompt-toolkit==3.0.43 protobuf==4.23.4 psutil @ file:///Users/cbousseau/work/recipes/ci_py311_2/psutil_1678995687212/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work pyarrow==15.0.0 pyarrow-hotfix==0.6 pyasn1==0.5.1 pyasn1-modules==0.3.0 pydantic==2.6.1 pydantic_core==2.16.2 pydub==0.25.1 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work PyJWT==2.8.0 pynvml==11.5.0 pyparsing==3.1.1 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work python-multipart==0.0.7 pytz==2023.4 PyYAML==6.0.1 pyzmq @ file:///private/var/folders/k1/30mswbxs7r1g6zwn8y4fyt500000gp/T/abs_43pxpbos3z/croot/pyzmq_1705605108344/work querystring-parser==1.2.4 referencing==0.33.0 regex==2023.12.25 requests==2.31.0 requests-oauthlib==1.3.1 responses==0.18.0 rich==13.7.0 rpds-py==0.17.1 rsa==4.9 s3fs==2023.10.0 safetensors==0.4.2 scikit-learn==1.2.2 scipy==1.12.0 semantic-version==2.10.0 sentencepiece==0.1.99 sentry-sdk==1.40.0 setproctitle==1.3.3 shortuuid==1.0.11 shtab==1.6.5 six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work smmap==5.0.1 sniffio==1.3.0 SQLAlchemy==2.0.25 sqlparse==0.4.4 stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work starlette==0.35.1 svgwrite==1.4.3 sympy==1.12 tabulate==0.9.0 tensorboard==2.15.1 tensorboard-data-server==0.7.2 termcolor==2.4.0 threadpoolctl==3.2.0 tiktoken==0.5.2 tokenizers==0.15.0 toolz==0.12.1 torch==2.2.0 torchaudio==2.2.0 torchvision==0.17.0 tornado @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_3a5nrn2jeh/croot/tornado_1696936974091/work tqdm==4.66.1 traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1704212992681/work transformers @ git+https://github.com/huggingface/transformers.git@bebeeee01275c32fccec3fa36d8b148d3813a7dc trl==0.7.10 typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1702176139754/work tyro==0.7.1 tzdata==2023.4 urllib3==2.0.7 uvicorn==0.27.0.post1 virtualenv==20.25.0 wandb==0.16.2 wavedrom==2.0.3.post3 wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1704731205417/work websockets==11.0.3 Werkzeug==3.0.1 wrapt==1.16.0 xxhash==3.4.1 yarl==1.9.4 zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work ```

Context for the issue:

Cannot use Outlines at all as it is on Mac

[lapp0]

Duplicate of #611

[lapp0]

@maximegmd my test suggests your issue is resolved by #556

You may try it early via pip install git+https://github.com/dtiarks/outlines@llama-logits-processor && pip install llama-cpp-python==0.2.37

[maximegmd]

Can confirm the PR solves the issues