[AutoParallel] Add auto parallel moe layer #9886
Conversation
|
Thanks for your contribution! |
Codecov ReportAttention: Patch coverage is
❌ Your patch check has failed because the patch coverage (14.18%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## develop #9886 +/- ##
===========================================
- Coverage 51.18% 50.91% -0.28%
===========================================
Files 745 748 +3
Lines 119016 119811 +795
===========================================
+ Hits 60924 60998 +74
- Misses 58092 58813 +721 ☔ View full report in Codecov by Sentry. |
| me = paddle.stack(me_list).mean(0) | ||
| ce = paddle.stack(ce_list).mean(0) | ||
| aux_loss = paddle.sum(me * ce) * float(self.num_experts) | ||
| return aux_loss |
There was a problem hiding this comment.
| # Make sure the capacity value does not exceed the number of tokens. | ||
| capacity = int(min(new_capacity, paddle.tensor(mask1.size(0)))) | ||
|
|
||
| l_aux = self._cal_aux_loss(gates, mask1) |
There was a problem hiding this comment.
pkuzyc
force-pushed
the
auto_moe_layer
branch
2 times, most recently
from
8bfa877 to
cbef5c5
Compare
pkuzyc
force-pushed
the
auto_moe_layer
branch
from
cbef5c5 to
143c203
Compare
| return mesh | ||
|
|
||
|
|
||
| def einsum(rule, a, b): |
| @@ -117,13 +117,13 @@ def scaled_dot_product_attention( | |||
| ) | |||
|
|
|||
| if isinstance(outputs, tuple): | |||
| outputs[0] = outputs[0].reshape([bsz, q_len, v_num_heads, head_dim]) | |||
| outputs[0] = outputs[0].reshape([bsz, kv_seq_len, v_num_heads, head_dim]) | |||
There was a problem hiding this comment.
Done. 这个函数里其他不一致的地方也改了
| # Make sure the capacity value does not exceed the number of tokens. | ||
| capacity = int(min(new_capacity, paddle.tensor(mask1.size(0)))) | ||
|
|
||
| l_aux = self._cal_aux_loss(gates, mask1) |
PR types
New features
PR changes
Models
Description
Add auto parallel moe layer