Circumvent Broken llama.cpp Pre-Tokenizer

[lapp0]

Fixes #820

Problem:

llama.cpp's pre-tokenizer doesn't handle unicode properly (draft PR: ggerganov/llama.cpp#5613). This results in tokens which are incompatible with Outlines byte-wise FSM and causes #820's error.

Solution:

• 1. If models.llamacpp() specifies a LlamaHFTokenizer, populate the vocabulary used in index construction with tokenizer.get_vocab(). This takes advantage of huggingface's working pre-tokenizer.
• 2. Warn users that they should pass a LlamaHFTokenizer:

>>> from outlines import models, generate
>>> model = models.llamacpp("Qwen/Qwen1.5-0.5B-Chat-GGUF", "*q8*.gguf")
/opt/conda/lib/python3.10/site-packages/outlines/models/llamacpp.py:294: UserWarning: llama.cpp pre-tokenizer is broken. You may recieve an Outlines error during Regex index construction.
To avoid this error when using `models.llamacpp` you may pass `tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained(<hf_repo_id>)` to `models.llamacpp()`
  warnings.warn(

Debug Notes / Observations:

• The problematic token in Qwen1.5 is 29333 (b' \xef\xbf\xbd')
• The issue can be reproduced with models.llamacpp, but not models.transformers
• AutoTokenizer's get_vocab() is inconsistent with its encode / decode output.
  • get_vocab()[ b'\xc4\xa0\xc3\xaf\xc2\xbf\xc2\xbd'.decode()] = 29333
  • get_vocab()[ b' \xef\xbf\xbd'.decode()] -> KeyError
  • encode( b'\xc4\xa0\xc3\xaf\xc2\xbf\xc2\xbd'.decode()) = [144242, 37572, 30182, 26062]
  • encode(b' \xef\xbf\xbd'.decode()) = [29333]

tokenizer.get_vocab() has a distinct mapping from tokenizer.decode due to the pre-tokenizer:

>>> tokenizer = transformers.AutoTokenizer.from_pretrained("Qwen/Qwen1.5-0.5B-Chat")
>>> tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(b' \xef\xbf\xbd'.decode())[0][0].encode()
b'\xc4\xa0\xc3\xaf\xc2\xbf\xc2\xbd'

[rlouf]

Looks good, thank you for the fix. I think it will be valuable to many people in the community 🙏