Improve Prefill Performance by Removing Redundant Padding and Optimizing Alltoall Communication

[SlightwindSec]

What this PR does / why we need it?

This PR improves Prefill performance by making two key optimizations:

1. Removing redundant padding before Flash Attention: This reduces unnecessary computation during attention operations.
2. Optimizing alltoall communication: The previous implementation involved one all_to_all_single call followed by three all_to_all calls. This has been refactored to use three all_to_all_single calls instead, with a fixed communication buffer to eliminate an extra communication step. This change not only simplifies the communication pattern but also leverages the better performance of all_to_all_single.

While there might be minor precision trade-offs, the choice of the coefficient 2 is an empirically sound value that maintains accuracy even when expert ID distribution is imbalanced.

In testing with DeepSeek-V3, the model was able to handle 3584-token inputs with significantly improved Prefill throughput and no regression in dialog quality.

Does this PR introduce any user-facing change?

No, this PR does not introduce any user-facing changes.

How was this patch tested?

• Verified correct generation behavior with DeepSeek-V3 model.
• Prefill performance was benchmarked with 3584-token inputs, showing noticeable speed improvements.
• Ensured that output quality remains consistent under typical workloads.