multi-gpu fused_moe tuning support #143
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
divakar-amd
changed the title
- todo: add fp8 support - todo: add comments and documentation
divakar-amd
force-pushed
the
distributed_fmoe_tuning
branch
from
cd68e07 to
37fc500
Compare
divakar-amd
requested review from
gshtras,
charlifu,
mawong-amd,
rasmith,
maleksan85,
hegemanjw4amd and
Alexei-V-Ivanov-AMD
|
retest |
gshtras
reviewed
Sep 20, 2024
vgokhale
reviewed
Oct 17, 2024
| # For now we see better perf with num_stages=0 for all gemm configs we care | ||
| # But keep this explicit so that we do not forget we may need to set it to | ||
| # other values in the future | ||
| num_stage_range = [0] |
There was a problem hiding this comment.
So this PR comes at a tricky time unfortunately.
See triton-lang/triton#4845 (review)
We are changing the sw pipelining such that it is more aligned with the nvidia side. After the above PR is merged (imminently), num_stages = 0 will actually be num_stages=2. If it is kept at 0, it will fail with an error as mentioned in the above link.
So we have two options
-
Hold this PR, if possible, until the Triton PR is submitted.
-
Submit this and then submit another once things break. As mentioned, the error message should be clear and should say what to do.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tuning_utils.pytorchrun benchmark_mixtral_moe_rocm.py --model 8x7B --modelTP 8 --numGPU 8