VLLM batch inference example is broken

[azgo14]

Describe the issue as clearly as possible:

The batch inference vllm examples don't run due to AttributeError returning.

In vllm 0.2.7 (also same w/ 0.2.6) there seems to be two interfaces. One is vllm.LLM and the other is the AsyncLLMEngine. Only AsyncLLMEngine has a tokenizer accessor which RegexLogitsProcessor and CFGLogitsProcessor assume.

To get batch inference to work I modify both of the above processors with something like this:

-        fsm = RegexFSM(regex_string, llm.tokenizer)
+        fsm = RegexFSM(regex_string, _adapt_tokenizer(llm.get_tokenizer()))

Steps/code to reproduce the bug:

Run `python examples/vllm_integration.py`

Expected result:

Normal execution, not a RuntimeError on accessor doesn't exist

Error message:

AttributeError: 'LLM' object has no attribute 'tokenizer'. Did you mean: 'get_tokenizer'?

Outlines/Python version information:

Head version of outlines. python3.10

Context for the issue:

No response

[rlouf]

We reverted a recent change that modified the vLLM integration. Could you retry this on the current version of main?

[sethkimmel3]

Hey @rlouf - this still appears to be broken. I'm getting AttributeError: 'LLM' object has no attribute 'tokenizer' error using the JSONLogitsProcessor. Using: vllm==0.3.0 and outlines==0.0.27.

Have you merged the above change into the most recent release?

[sethkimmel3]

I got this to work by explicitly calling the adapt_tokenizer method found here. However, I'm pretty confused why this is necessary given that the JSONLogitsProcessor seems to instantiate the RegexLogitsProcessor.

I also needed to include the _patched_apply_logits_processors, as shown in this issue comment.

[lapp0]

@sethkimmel3 could you please create an issue and provide a reproduction script? I think the docstring below is inaccurate, it actually seems to take an _AsyncLLMEngine

https://github.com/outlines-dev/outlines/blob/7fae436345e621a955e1e6ea610f74cf59f9466f/outlines/serve/vllm.py#L42-L53

[sethkimmel3]

Done @lapp0 - #624