-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add test for open-llama-3b-v2-f16 model through sharktank. #272
Conversation
95c1cab
to
5fc416a
Compare
Nice, is there a reason that sharktank needs its own config file? Pytorch models and sharktank have the same starting point of mlir, so maybe just having one overarching models configuration could work. |
I'm going back and forth on that, thanks for noticing too. With multiple files we keep the test lists separate. The lists are short, but unqualified right now. I think I want them to be qualified for merging, so this: SHARK-TestSuite/iree_tests/configs/config_pytorch_models_cpu_llvm_task.json Lines 10 to 14 in 4751fab
would be "skip_compile_tests": [
"pytorch/models/sdxl-scheduled-unet-3-tank",
"pytorch/models/sdxl-vae-decode-tank",
"pytorch/models/sdxl-prompt-encoder-tank"
], then we'd also have new models: "skip_compile_tests": [
"pytorch/models/sdxl-scheduled-unet-3-tank",
"pytorch/models/sdxl-vae-decode-tank",
"pytorch/models/sdxl-prompt-encoder-tank",
"sharktank/llama/open-llama-3b-v2-f16",
], Let me check if that works... the test names are still a bit awkwardly passed through pytest / conftest.py. |
I'm also debating naming/grouping PR currently:
One alternative (grouping ops across frameworks and models across frameworks):
|
... and deciding how to run the tests: By directory, reusing configs:
By directory, separate configs:
If we ran Ah! Nevermind -- we can run |
I think of the options, by directory reusing configs looks like the cleanest |
Sorry, building up a stack of merge conflicts with #271 here x_x |
Splitting off a few smaller PRs from this: |
688c6bb
to
fe8d3bc
Compare
fe8d3bc
to
332f50f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Progress on nod-ai/sharktank#22. See nod-ai/SHARK-TestSuite#272 for the specifics of what the new test is exercising. The "models" tests now include `pytorch/models/` and `sharktank/`, so all test names are qualified relative to `iree_tests/` in the test suite repo. (Totally inflating my commit stats here, sorry :P) ci-exactly: build_packages,regression_test
Progress on nod-ai/sharktank#22 This adds one test for a llama model running through https://github.com/nod-ai/sharktank. That project is still getting set up, so new docs for this particular workflow are coming in at nod-ai/sharktank#69 and tests in that repo are in nod-ai/sharktank#70. Specifically, this exercises: * [`sharktank/models/llama/llama.py`](https://github.com/nod-ai/sharktank/blob/main/sharktank/sharktank/models/llama/llama.py) * [`sharktank/examples/export_paged_llm_v1.py`](https://github.com/nod-ai/sharktank/blob/main/sharktank/sharktank/examples/export_paged_llm_v1.py) with batch sizes == [4] * The `open-llama-3b-v2-f16.gguf` file from https://huggingface.co/SlyEcho/open_llama_3b_v2_gguf * Compilation and crashless execution, _not_ numerical correctness (yet) Ideas for future work: * Test cases for the same model/parameters * Other batch sizes * `decode()` as well as `prefill()` * Real inputs with expected outputs (`decode()` crashes on some faked inputs still 🤔) * Other flag combinations and target configurations (starting simple though) * Test cases for other models/parameters * 8b / 70b parameter models * Mistral, Mixtral, Gemma, etc.
Progress on nod-ai/sharktank#22. See nod-ai/SHARK-TestSuite#272 for the specifics of what the new test is exercising. The "models" tests now include `pytorch/models/` and `sharktank/`, so all test names are qualified relative to `iree_tests/` in the test suite repo. (Totally inflating my commit stats here, sorry :P) ci-exactly: build_packages,regression_test Signed-off-by: Lubo Litchev <[email protected]>
Progress on nod-ai/sharktank#22
This adds one test for a llama model running through https://github.com/nod-ai/sharktank. That project is still getting set up, so new docs for this particular workflow are coming in at nod-ai/sharktank#69 and tests in that repo are in nod-ai/sharktank#70.
Specifically, this exercises:
sharktank/models/llama/llama.py
sharktank/examples/export_paged_llm_v1.py
with batch sizes == [4]open-llama-3b-v2-f16.gguf
file from https://huggingface.co/SlyEcho/open_llama_3b_v2_ggufIdeas for future work:
decode()
as well asprefill()
decode()
crashes on some faked inputs still 🤔)