Invalid generate.json Output with models.llamacpp

[lapp0]

Related:

• Validation Error during pydantic validation for Llama3 GGUF #952
• Add a non_strict mode to deal with JSONDecodeError errors #985

Describe the issue as clearly as possible:

models.llamacpp used with generate.json can result in an output which isn't valid with the pydantic schema.

Steps/code to reproduce the bug:

# Reproducer from "Robert Roland Roger" on Discord:

import outlines
from pydantic import BaseModel, StringConstraints
from typing import Annotated
import llama_cpp

model = outlines.models.llamacpp(
    "TheBloke/zephyr-7B-beta-GGUF", 
    "zephyr-7b-beta.Q6_K.gguf",
    tokenizer=llama_cpp.llama_tokenizer.LlamaHFTokenizer.from_pretrained("HuggingFaceH4/zephyr-7b-beta")
)

class Character(BaseModel):
    name: Annotated[str, StringConstraints(max_length=10)]
    #age: int = Field(..., ge=18, lt=99)
    age: int
    #strength: int = Field(..., ge=1, lt=100)
    strength: int

generator = outlines.generate.json(model, Character)

prompt = """
<|system|>
You always output valid JSON.
<|user|>
Generate a new character in valid json. Use the following fields:
name, age (between 18 and 99), armor beween 1 and 100) and strength.
<|assistant|>
"""
character = generator(prompt)

Expected result:

An output which pydantic recognizes as valid with the schema.

Error message:

Expecting ',' delimiter: line 1 column 28 (char 27) [type=value_error.jsondecode, input_value='{ "name": "Elena", "age": 2', input_type=str]

Outlines/Python version information:

Version information

``` outlines==0.0.44 llama-cpp-python==0.2.78 ```

Another Reproducer using models.transformers_multimodal

import outlines

model = outlines.models.transformers_multimodal(
    "llava-hf/llava-v1.6-mistral-7b-hf",
	device="cuda",
	model_kwargs=dict(torch_dtype=torch.bfloat16)
)

from pydantic import BaseModel
from typing import List, Optional

class ImageData(BaseModel):
    caption: str
    tags: List[str]
    location: Optional[str]
    objects: List[str]

image_data_generator = outlines.generate.json(model, ImageData)
image_data_generator(
    "<image> detailed JSON metadata:",
    "https://upload.wikimedia.org/wikipedia/commons/e/ea/FCAB_EMD_GT22CU-3_San_Pedro_-_Ascotan.jpg"
)