[Bugfix] Fix CFGGuide and use outlines for grammars that can't convert to GBNF #11389
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: mgoin <[email protected]>
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
Signed-off-by: mgoin <[email protected]>
mgoin
requested review from
DarkLight1337,
robertgshaw2-redhat and
simon-mo
as code owners
Signed-off-by: mgoin <[email protected]>
mgoin
changed the title
mgoin
changed the title
aarnphm
approved these changes
Dec 21, 2024
DarkLight1337
approved these changes
Dec 23, 2024
ayylemao
pushed a commit
to ayylemao/vllm
that referenced
this pull request
Dec 24, 2024
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]>
ZenPuzzle
pushed a commit
to ZenPuzzle/vllm
that referenced
this pull request
Dec 25, 2024
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]>
BKitor
pushed a commit
to BKitor/vllm
that referenced
this pull request
Dec 30, 2024
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]>
joennlae
pushed a commit
to 44ai-labs/vllm
that referenced
this pull request
Jan 19, 2025
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]>
joennlae
pushed a commit
to 44ai-labs/vllm
that referenced
this pull request
Jan 19, 2025
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]>
abmfy
pushed a commit
to abmfy/vllm-flashinfer
that referenced
this pull request
Jan 24, 2025
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]> Signed-off-by: Bowen Wang <[email protected]>
abmfy
pushed a commit
to abmfy/vllm-flashinfer
that referenced
this pull request
Jan 24, 2025
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]>
mgawarkiewicz
pushed a commit
to HabanaAI/vllm-fork
that referenced
this pull request
Feb 12, 2025
This PR mostly ports vllm-project#11389 to the design introduced by #358 and makes the custom caching code a little bit more robust. Currently there are two problems with guided decode: - `mask[list(allowed_tokens)] = 0` is causing crashes due to allowed_tokens containing tensors. Pretty easy fix. - The value type of `self._fsm_state` was changed from `int` to union of `int` and `outlines.state.CFGState`, which may cause `self._cached_get_mask_tensor(state_id, scores.size(-1), scores.device)` to crash, as `outlines.state.CFGState` is not hashable. This PR changes the caching mechanism so that if function arguments are not hashable, their id is taken as key. This might cause some cache misses, but that's better than crashing, as it does right now. None of the above is problem on upstream, as this stems from code introduced in #358. I've also added guided decode tests to CI suite.
lk-chen
pushed a commit
to lk-chen/vllm
that referenced
this pull request
Mar 5, 2025
…t to GBNF (vllm-project#11389) Signed-off-by: mgoin <[email protected]> Signed-off-by: Linkun Chen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@afeldman-nm found this issue while trying the offline and online structured output examples like
python examples/offline_inference_structured_outputs.py. We were failing to convert the sql lark grammar into gbnf format for xgrammar and had that check+conversion in the wrong place to fallback to another backend.It turns out that our outlines backend actually regressed on this example as well with the upgrade to 0.1.11 (it works fine if I use
outlines==0.0.46). I made this PR to make xgrammar fallback to outlines in this case, and get outlines back to functional for it.