Batched generation does not work

[maximilianmordig]

Describe the issue as clearly as possible:

Perform batched generation. The problem occurs because CFGGuide.copy() which does not copy the attribute regex_fsm. https://github.com/outlines-dev/outlines/blob/25b6bcd498d0f335683f0026e9baeb591061d001/outlines/fsm/guide.py#L432

Steps/code to reproduce the bug:

from outlines import models, generate

grammar = """?start: "a".."z" """
model = models.transformers("gpt2")
generator = generate.cfg(model, grammar)
sequence = generator("hello")
sequence = generator(["hello", "world"])
print(sequence)

Expected result:

Batched generations

Error message:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
File /somepath/scratch.py:2
      1 #%%
----> 2 sequence = generator(["hello", "world"])
      3 print(sequence)

File /someenv//lib/python3.10/site-packages/outlines/generate/api.py:207, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng)
    205 while True:
    206     try:
--> 207         last_state = next(states)
    208         if max_tokens or stop_sequences:
    209             token_ids = last_state.token_ids

File /someenv//lib/python3.10/site-packages/outlines/generate/generator.py:93, in sequence_generator(model, sampler, fsms, token_ids, sequence_weights, attention_masks, fsm_states, rng)
     90     fsms = reorder_fsms(fsms, ancestors)
     91     fsm_states = reorder_fsm_states(fsm_states, ancestors)
---> 93 fsm_states = get_next_fsm_states(fsms, fsm_states, next_token_ids)
     94 is_finished = is_generation_finished(fsms, fsm_states)
     96 if is_finished:

File /someenv//lib/python3.10/site-packages/outlines/generate/generator.py:132, in get_next_fsm_states(fsms, fsm_states, next_token_ids)
    115 def get_next_fsm_states(
    116     fsms: List["Guide"], fsm_states: List[int], next_token_ids: "torch.Tensor"
    117 ) -> List[int]:
    118     """
    119 
    120     Parameters
   (...)
    130 
    131     """
--> 132     return [
    133         fsm.get_next_state(fsm_state, int(token_id[0]))
    134         for fsm, fsm_state, token_id in zip(fsms, fsm_states, next_token_ids)
    135     ]

File /someenv//lib/python3.10/site-packages/outlines/generate/generator.py:133, in <listcomp>(.0)
    115 def get_next_fsm_states(
    116     fsms: List["Guide"], fsm_states: List[int], next_token_ids: "torch.Tensor"
    117 ) -> List[int]:
    118     """
    119 
    120     Parameters
   (...)
    130 
    131     """
    132     return [
--> 133         fsm.get_next_state(fsm_state, int(token_id[0]))
    134         for fsm, fsm_state, token_id in zip(fsms, fsm_states, next_token_ids)
    135     ]

File /someenv//lib/python3.10/site-packages/outlines/fsm/guide.py:419, in CFGGuide.get_next_state(self, state, token_id)
    416     self.reset_state = False
    417     state = self.start_state
--> 419 return self.regex_fsm.get_next_state(state, token_id)

AttributeError: 'CFGGuide' object has no attribute 'regex_fsm'

Outlines/Python version information:

0.0.46
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
accelerate==0.31.0
aiohttp==3.9.5
aiosignal==1.3.1
annotated-types==0.7.0
asttokens==2.4.1
async-timeout==4.0.3
attrs==23.2.0
bitsandbytes==0.43.1
certifi==2024.6.2
cfgv==3.4.0
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==3.0.0
comm==0.2.2
contourpy==1.2.1
cycler==0.12.1
datasets==2.20.0
debugpy==1.8.2
decorator==5.1.1
dill==0.3.8
diskcache==5.6.3
distlib==0.3.8
docker-pycreds==0.4.0
docstring_parser==0.16
einops==0.8.0
exceptiongroup==1.2.1
executing==2.0.1
filelock==3.15.4
flake8==7.1.0
flash-attn==2.5.9.post1
fonttools==4.53.0
frozenlist==1.4.1
fsspec==2024.5.0
gitdb==4.0.11
GitPython==3.1.43
huggingface-hub==0.23.4
identify==2.5.36
idna==3.7
iniconfig==2.0.0
interegular==0.3.3
ipykernel==6.29.5
ipython==8.26.0
jedi==0.19.1
Jinja2==3.1.4
jsonschema==4.22.0
jsonschema-specifications==2023.12.1
jupyter_client==8.6.2
jupyter_core==5.7.2
kiwisolver==1.4.5
lark==1.1.9
llvmlite==0.43.0
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.9.0
matplotlib-inline==0.1.7
mccabe==0.7.0
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
nest-asyncio==1.6.0
networkx==3.3
nodeenv==1.9.1
numba==0.60.0
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.5.82
nvidia-nvtx-cu12==12.1.105
outlines==0.0.46
packaging==24.1
pandas==2.2.2
parso==0.8.4
peft==0.11.1
pexpect==4.9.0
pillow==10.4.0
platformdirs==4.2.2
pluggy==1.5.0
pre-commit==3.7.1
prompt_toolkit==3.0.47
protobuf==5.27.2
psutil==6.0.0
ptyprocess==0.7.0
pure-eval==0.2.2
pyairports==2.1.1
pyarrow==16.1.0
pyarrow-hotfix==0.6
pycodestyle==2.12.0
pycountry==24.6.1
pydantic==2.8.2
pydantic_core==2.20.1
pyflakes==3.2.0
Pygments==2.18.0
pyparsing==3.1.2
pytest==8.2.2
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
pyzmq==26.0.3
referencing==0.35.1
regex==2024.5.15
requests==2.32.3
rich==13.7.1
rpds-py==0.18.1
safetensors==0.4.3
sentry-sdk==2.7.1
setproctitle==1.3.3
shtab==1.7.1
six==1.16.0
smmap==5.0.1
stack-data==0.6.3
sympy==1.12.1
tokenizers==0.19.1
tomli==2.0.1
torch==2.3.1
torchaudio==2.3.1
torchvision==0.18.1
tornado==6.4.1
tqdm==4.66.4
traitlets==5.14.3
transformers==4.42.3
triton==2.3.1
trl==0.9.4
typing_extensions==4.12.2
tyro==0.8.5
tzdata==2024.1
urllib3==2.2.2
virtualenv==20.26.3
wandb==0.17.3
wcwidth==0.2.13
xxhash==3.4.1
yarl==1.9.4

Context for the issue:

Cannot use batching with cfg

[maximilianmordig]

Temporary fix:

from outlines.fsm.guide import CFGGuide
def new_copy(self):
    """Create a copy of the FSM."""
    res = CFGGuide(self.cfg_string, self.tokenizer)
    if hasattr(self, "regex_fsm"):
        res.regex_fsm = self.regex_fsm.copy()
    return res
CFGGuide.copy = new_copy

However, it does not seem to work since the output is much shorter when generating the batch.

[lapp0]

@maximilianmordig can you please try again main since the following PR has merged?

#1067