Guided decoding with xgrammar

[windreamer]

Motivation

LMDeploy’s TurboMind backend is the fastest inference stack in the ecosystem, yet it still lacks Guided Decoding – a feature that is already available in the PyTorch backend and heavily requested by the community.
This PR closes the gap by bringing token-level, C++ native Guided Decoding to TurboMind while keeping the API 100 % compatible with the existing PyTorch backend.
The implementation is built on xGrammar (Apache-2.0), a high-performance C++ library that compiles JSON / Choice / Regex grammars into token FSMs and applies them with negligible overhead.

Modification

1. Build-system

   • Add xgrammar as a header-only dependency via CMake FetchContent (CUDA & Python bindings disabled).
   • Export xgrammar::tokenizer_info and xgrammar::grammar_compiler symbols under lmdeploy::xgrammar.

2. Core C++ changes

   • DynamicDecodeLayer pipeline extended with two new layers:
     • GuidedDecodeMaskLayer: in setup() compiles / reuses grammar → builds per-request token bitmask; in forward() launches a light CUDA kernel to mask disallowed logits to -INF.
     • GuidedDecodeUpdateLayer: in forward() calls matcher->AcceptToken(output_id) to advance the FSM.
   • Grammar compiler cache (LRU, keyed by schema hash) shared across all sessions to avoid re-compilation.

3. Python frontend

   • Re-use existing guided_decoding utilities from PyTorch backend; no new API surface.
   • turbo.TurboMindEngine now accepts the same response_format= / guided_json= / guided_choice= arguments.

Checklist

• Pre-commit hooks (clang-format, flake8, mypy) passed.
• Document updated

[shell-nlp]

good job!

[lvhan028]

The sampling-related classes are declared as templates, but the template parameter T does not appear to be utilized in any of the following:

• Member variable
• Member function
• Base class
  how about remove the templates? @lzhangzz @irexyc any comments?

[lvhan028]

cc @zhulinJulia24 may consider CI for guided decoding functions.
Here is the guide https://lmdeploy.readthedocs.io/en/latest/advance/structed_output.html

[windreamer]

What's the size of whl file if this PR is applied?

 ls -alh lmdeploy-0.10.0-cp310-cp310-linux_x86_64.whl
-rw-rw-r-- 1 tianzhongbo tianzhongbo 92M Sep 25 09:07 lmdeploy-0.10.0-cp310-cp310-linux_x86_64.whl

It seems we will increase the package size from 79M to 92M, which is 17%

[windreamer]

Also done replace outdated outlines with xgrammar in PyTorch Engine. And this enabled us to:

• No longer need to restrict numpy<2
• No buggy pyairports packages as dependencies