Skip to content

Commit

Permalink
[torch.compile] add deepseek v2 compile (vllm-project#9775)
Browse files Browse the repository at this point in the history
Signed-off-by: youkaichao <[email protected]>
  • Loading branch information
youkaichao authored Oct 28, 2024
1 parent 97b61bf commit 76ed534
Showing 1 changed file with 2 additions and 0 deletions.
2 changes: 2 additions & 0 deletions vllm/model_executor/models/deepseek_v2.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
from transformers import PretrainedConfig

from vllm.attention import Attention, AttentionMetadata
from vllm.compilation.decorators import support_torch_compile
from vllm.config import CacheConfig
from vllm.distributed import (get_pp_group,
get_tensor_model_parallel_world_size,
Expand Down Expand Up @@ -403,6 +404,7 @@ def forward(
return hidden_states, residual


@support_torch_compile
class DeepseekV2Model(nn.Module):

fall_back_to_pt_during_load = False
Expand Down

0 comments on commit 76ed534

Please sign in to comment.