Simulate how llm serving engines like vllm make use of python asyncio.Queue to achieve dynamic batching: batch generation at the iteration level.
hitpoint6/llm-continuous-batching-simulator
Folders and files
| Name | Name | Last commit date | ||
|---|---|---|---|---|
Repository files navigation