Add test script and workflow for llama export, compile, serve. #70
Conversation
tests/llama_export_compile_serve.sh
Outdated
There was a problem hiding this comment.
Is https://github.com/nod-ai/sharktank/tree/main/sharktank/sharktank/models/punet running through IREE yet? We could set up a similar set of tests for that model too.
There was a problem hiding this comment.
Yes, but fragile. Can start with just eager and import/export. Need to fetch from an HF repo to do it.
tests/llama_export_compile_serve.sh
Outdated
There was a problem hiding this comment.
Yes, but fragile. Can start with just eager and import/export. Need to fetch from an HF repo to do it.
Good to know for punet. In other repos (IREE, SHARK-TestSuite) we have a few persistent runners and test scripts are already set up to use a local cache. This is currently using a 7GB .gguf file (3b parameters in f16) from an open model on huggingface. We can test with other quantized data types and larger parameter sizes too. That may work well enough on nightly runs using standard runners but yeah - should use larger runners with persistent caches for best results. |
Progress on nod-ai/sharktank#22 This adds one test for a llama model running through https://github.com/nod-ai/sharktank. That project is still getting set up, so new docs for this particular workflow are coming in at nod-ai/sharktank#69 and tests in that repo are in nod-ai/sharktank#70. Specifically, this exercises: * [`sharktank/models/llama/llama.py`](https://github.com/nod-ai/sharktank/blob/main/sharktank/sharktank/models/llama/llama.py) * [`sharktank/examples/export_paged_llm_v1.py`](https://github.com/nod-ai/sharktank/blob/main/sharktank/sharktank/examples/export_paged_llm_v1.py) with batch sizes == [4] * The `open-llama-3b-v2-f16.gguf` file from https://huggingface.co/SlyEcho/open_llama_3b_v2_gguf * Compilation and crashless execution, _not_ numerical correctness (yet) Ideas for future work: * Test cases for the same model/parameters * Other batch sizes * `decode()` as well as `prefill()` * Real inputs with expected outputs (`decode()` crashes on some faked inputs still 🤔) * Other flag combinations and target configurations (starting simple though) * Test cases for other models/parameters * 8b / 70b parameter models * Mistral, Mixtral, Gemma, etc.
Progress on nod-ai/sharktank#22 This adds one test for a llama model running through https://github.com/nod-ai/sharktank. That project is still getting set up, so new docs for this particular workflow are coming in at nod-ai/sharktank#69 and tests in that repo are in nod-ai/sharktank#70. Specifically, this exercises: * [`sharktank/models/llama/llama.py`](https://github.com/nod-ai/sharktank/blob/main/sharktank/sharktank/models/llama/llama.py) * [`sharktank/examples/export_paged_llm_v1.py`](https://github.com/nod-ai/sharktank/blob/main/sharktank/sharktank/examples/export_paged_llm_v1.py) with batch sizes == [4] * The `open-llama-3b-v2-f16.gguf` file from https://huggingface.co/SlyEcho/open_llama_3b_v2_gguf * Compilation and crashless execution, _not_ numerical correctness (yet) Ideas for future work: * Test cases for the same model/parameters * Other batch sizes * `decode()` as well as `prefill()` * Real inputs with expected outputs (`decode()` crashes on some faked inputs still 🤔) * Other flag combinations and target configurations (starting simple though) * Test cases for other models/parameters * 8b / 70b parameter models * Mistral, Mixtral, Gemma, etc.
Progress on #22
Sample runs on my fork:
I decided to run this on a nightly
scheduleand onworkflow_dispatch. It takes around 10 minutes so it could run onpull_requestif we want too.As these components stabilize and we spend less time hacking on individual steps using the full toolkit (python -> manual
iree-compilevs. using the in-process compiler API) we can switch the test from a bash script to a pytest file. Need to start somewhere :)