Support for DeepseekV2ForCausalLM

[tgandrew]

Question

Can we get DeepseekV2 supported?

Code to reproduce

from tensorrt_llm import LLM, SamplingParams


prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="deepseek-ai/DeepSeek-Coder-V2-Lite-Base", trust_remote_code=True)

outputs = llm.generate(prompts, sampling_params)

# Print the outputs.
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

Error

Traceback (most recent call last):
  File "/workspace/test.py", line 13, in <module>
    llm = LLM(model="deepseek-ai/DeepSeek-Coder-V2-Lite-Base", trust_remote_code=True)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm.py", line 152, in __init__
    raise e
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm.py", line 147, in __init__
    self._build_model()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm.py", line 310, in _build_model
    self._engine_dir = model_loader()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 1318, in __call__
    return self._build_model()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 1434, in _build_model
    build_task(self.get_engine_dir())
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 1403, in build_task
    model_loader(engine_dir=engine_dir)
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 1000, in __call__
    pipeline()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 942, in __call__
    self.step_forward()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 971, in step_forward
    self.step_handlers[self.counter]()
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/hlapi/llm_utils.py", line 1116, in _load_model_from_hf
    model_cls = AutoModelForCausalLM.get_trtllm_model_class(
  File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/automodel.py", line 54, in get_trtllm_model_class
    raise NotImplementedError(
NotImplementedError: The given huggingface model architecture DeepseekV2ForCausalLM is not supported in TRT-LLM yet

[fengyang95]

@tgandrew FYI: #1758

[dominicshanshan]

@fengyang95, little update as promised: CI pass and code review ongoing, will add some benchmark data and analysis in README, MR is pretty huge so took little longer time, will update the status on Monday, thanks for support TRT-LLM !

[fengyang95]

@fengyang95, little update as promised: CI pass and code review ongoing, will add some benchmark data and analysis in README, MR is pretty huge so took little longer time, will update the status on Monday, thanks for support TRT-LLM !

nice work!!!

[dominicshanshan]

little update on Friday: we are in code rebase and possible merge in main branch in next week... finger crossed.. thanks!

[dominicshanshan]

@fengyang95 , thanks for attention on Tensorrt-LLM, deepseek-v2 is live in main branch and will be officially announced in v0.15, will close this bug now .. welcome to give any suggestion or issue when using Tensorrt-LLM..

[fengyang95]

@fengyang95 , thanks for attention on Tensorrt-LLM, deepseek-v2 is live in main branch and will be officially announced in v0.15, will close this bug now .. welcome to give any suggestion or issue when using Tensorrt-LLM..

@dominicshanshan Nice work!!! Looking forward to support for fp8 and cc8.9 (e.g., L40) since our GPU resources are relatively limited.

[dominicshanshan]

@fengyang95 , thanks for attention on Tensorrt-LLM, deepseek-v2 is live in main branch and will be officially announced in v0.15, will close this bug now .. welcome to give any suggestion or issue when using Tensorrt-LLM..

@dominicshanshan Nice work!!! Looking forward to support for fp8 and cc8.9 (e.g., L40) since our GPU resources are relatively limited.

message received, working on FP8 and sm89, sm89 support should available soon, will update the status once it passed our internal tests..

[Pernekhan]

@dominicshanshan really appreciate your work on deepseek.

is the fp8 support getting closer?

[dominicshanshan]

@dominicshanshan really appreciate your work on deepseek.

is the fp8 support getting closer?

yes, working on it, should be enable soon..