Onboard DeepSeek MoE with shared experts

[RissyRan]

Description

Onboard DeepSeek MoE with shared experts (functional first - reference from DeepSeek):

• Refactor models.py to handle mixed layers, i.e. both dense and moe layers
• Add DeepSeek v3 config and deepseek.py as decoder layer
• Add DeepSeekMoeBlock to reuse the dense and moe blocks so we could use either dropping or dropless for future tuning
• Add a compile test

Tests

base_num_decoder_layers: 5 & num_experts: 16

• Small config - Functional tests for scan_layers=True: link
• Small config - Functional tests for scan_layers=False: link
• One profile- matmul shapes on both dense and moe LGTM

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.

[gagika]

Only a few minor comments, Thanks!

[gagika]

Can you comment what are main differences of DeepSeekMoELayer from regular MoELayer?

[gagika]

Can you add some comment how/why it's different for deepseek?

also, perhaps you can move it to a function, as it's used in 2 places, e.g.

def _deepseek_scale_weights(self, weights):
"""Scales weights according to DeepSeek's ... ."""
weights /= weights.sum(-1, keepdims=True)
weights *= self.config.routed_scaling_factor
return weights