Now I use outlines+vllm and it takes 10 seconds to generate a JSON. This speed is not feasible in my current business. Is there any accelerated solution, such as a paper, plan or PR?

[lwdnxu]

Now I use outlines+vllm and it takes 10 seconds to generate a JSON. This speed is not feasible in my current business. Is there any accelerated solution, such as a paper, plan or PR?

[cpfiffer]

Speed improvements are always being worked on, but it may also be worth discussing your general setup. What models and schemas do you tend to work with? There might be some places we can suggest performance improvements.

[lwdnxu]

model： qwen2-7b
GPU: 4090
I want to generate json like this :{
"dataSource": "1002",
"column": "200-279",
"value": [
"360"
],
"mark": "16",
"unit": null
}
Number of tokens per request: 4166
Using vllm+ outlines, the speed is about 10+ seconds
Using vllm, the speed is about 1.x seconds

I would like to ask if there is an optimized solution? And what is the maximum number of seconds you can achieve in this case?

[lapp0]

@lwdnxu Each new schema requires an index to be compiled, which has some overhead. Once generation starts, overhead is ~1ms/token. This index only needs to be compiled once per unique schema.

Are you using a new schema each time, or are you seeing slowness re-using the same schema?