GptManager occupies the entire CPU

[Masterlk]

I use pybings of tensorrt-llm, and try with GptManager for InflightBatching feature. but it's seems the initialization of GptManager occupies the entire CPU, and it hasn't even accepted a request yet.

My expectation is that when there are no requests, the CPU utilization should be 0. so what's the possible reason?

I use the main branch, the newest version, and this code can reproduce the problem：

import argparse
from asyncio import run
from pathlib import Path
import asyncio
from tensorrt_llm.engine import AsyncLLMEngine

async def main(model_dir, tokenizer_dir):
    engine = AsyncLLMEngine(model_dir, tokenizer_dir)
    await asyncio.sleep(100)


if __name__ == "__main__":
    model_dir = "./1-gpu"
    tokenizer_dir = "../module/mistral/data/tokenizer"
    run(main(model_dir, tokenizer_dir))

[byshiue]

Does the program occupy all threads or only single thread? GptManager will create a thread to wait and collect the requests.

[Masterlk]

Does the program occupy all threads or only single thread? GptManager will create a thread to wait and collect the requests.

I use top -Hp xxx to see all threads of the process, and found a single thread occupy the cpu to 100%

[byshiue]

Then I think it is expected.

[Masterlk]

Then I think it is expected.

If the thread only wait and collect the requests, then why it occupy the entire cpu core ? Theoretically, if no requests come in, waiting should not occupy CPU resources

[byshiue]

It uses a while loop to wait the request coming.

[lyc728]

pybings

import argparse
from asyncio import run
from pathlib import Path
import asyncio
from tensorrt_llm.engine import AsyncLLMEngine

async def main(model_dir, tokenizer_dir):
engine = AsyncLLMEngine(model_dir, tokenizer_dir)
await asyncio.sleep(100)

if name == "main":
model_dir = "./1-gpu"
tokenizer_dir = "../module/mistral/data/tokenizer"
run(main(model_dir, tokenizer_dir))

Hello, do you have the complete code? I don't see InflightBatching implemented by tensorrt-llm. May I ask where is your code?

[HUSTHY]

pybings

import argparse from asyncio import run from pathlib import Path import asyncio from tensorrt_llm.engine import AsyncLLMEngine

async def main(model_dir, tokenizer_dir): engine = AsyncLLMEngine(model_dir, tokenizer_dir) await asyncio.sleep(100)

if name == "main": model_dir = "./1-gpu" tokenizer_dir = "../module/mistral/data/tokenizer" run(main(model_dir, tokenizer_dir))

Hello, do you have the complete code? I don't see InflightBatching implemented by tensorrt-llm. May I ask where is your code?

the InflightBatching is implemented when built trt engine and for AsyncLLMEngine use

[nv-guomingz]

Hi @Masterlk do u still have further issue or question now? If not, we'll close it soon.

[HUSTHY]

已收到！谢谢！                          ——黄洋

[HUSTHY]

已收到！谢谢！                          ——黄洋