Benchmark Llama 3.1 f16 and fp8 with CI

[aviator19941]

Adds pytests for f16 and fp8 with the CI. Currently Llama 3.1 8B f16 is the only test that fully benchmarks through. Llama 3.1 8B fp8, Llama 3.1 70B f16/fp8, and Llama 3.1 405B f16/fp8 tests are marked as XFAIL for now.

[IanNod]

output files should be temporary. We don't want them lasting or being picked up accidentally on subsequent runs

[aviator19941]

Added a function to remove the output files at the end of each test

[ScottTodd]

This can be separate from this PR, but we should use an in-memory API for all model exporting/importing workflows, rather than fork into a subprocess.

Users shouldn't need to chain together python -m sharktank.examples. and iree-compile ... commands. We can aim for something like https://docs.vllm.ai/en/latest/getting_started/quickstart.html#offline-batched-inference

llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)

(that's as minimal as it gets - we'll want to pass options like the compilation target though)

[aviator19941]

Makes sense, I'll try to update it in the next PR

[archana-ramalingam]

Updating the branch with changes from main will fix the second CI test.

[IanNod]

probably better to use tempfile like done here https://github.com/nod-ai/SHARK-Platform/blob/0c2e965c3ffe723db2fe2be9193c6d45fe558dbe/sharktank/tests/types/tensors_test.py#L71. Should better handle potential multiple async runs effectively corrupting the data in these effectively hardcoded paths

[aviator19941]

We want to save the output files to track any regressions right? So, do we still need tempfile? I think os.mkdir() would make more sense to use in this case, especially when specifying the directory path of the artifiacts to upload.

[IanNod]

you can upload the output artifacts before tempfile gets rid of them. I don't think we want to keep the files on the CI system permanently. os.mkdir could still have issues with multiple simultaneous instances running interfering with each other unless you are very careful with it

[aviator19941]

How can I upload the output artifacts before tempfile gets rid of them?

[IanNod]

https://github.com/nod-ai/SHARK-TestSuite/blob/95b35cc076618ef79b7f84dc4117f4528451e5e6/e2eshark/_run_helper.py#L61 for example uploading files. You should just need to do that before closing the tempfile or directory that is created

[saienduri]

Do we prefer sharkpublic? We will be uploading a bunch of artifacts there every night, and it might be hard to correlate which run correlates to which files. With github actions, we can just click on a run and see it's respective output artifacts without cluttering the storage

[IanNod]

Does github actions work well for downloading things like vmfbs? If so, then that would work well.

[saienduri]

I haven't tried uploading vmfbs before as artifacts, but we can check before merging

[ScottTodd]

How much data are we talking about here? Cloud storage isn't cheap. Using GitHub artifacts with GitHub workflows is nice since it doesn't require separate privileges to maintain, but then you're limited by GitHub's retention policies.

If you want to keep historical data in an organized way, please design a database early. I've seen too many benchmarking systems without that structure and it makes analysis tricky.

For this PR itself, I'd try to keep the scope small and checkpoint the work, rather than get hung up on all these discussions. Fine to land and iterate a bit so things don't pile up.

[saienduri]

Yeah let's start with github artifacts without a temp dir (it won't persist between runs anyways) and iterate from there

[IanNod]

why skip instead of xfail?

[aviator19941]

Because the test is still able to run successfully since it is still able to export an IR and compile and benchmark fully through, however, it is not exporting the correct IR yet. The attention_kernel support needs to be added for it to execute as expected. If I mark the test as XFAIL, it will still output benchmark numbers for 8B f16 decomposed even though we want the non-decomposed numbers in this test.

[IanNod]

We should raise an error when calling the attention_kernel flag with torch_sdpa until that is plumbed through and then we can xfail here

[IanNod]

To more easily handle regressions and such I think we really want to upload and store the model IR, vmfb, and dispatch dump so we can triage exactly what this benchmark used.

[aviator19941]

Makes sense, I will save the temporary files into the github actions file then.

[IanNod]

I was kind of thinking of just having it upload to sharkpublic since that would be easy to access for those who want them. Not sure if github actions file would be better or not

[IanNod]

likely need at least a comment that this is system specific for A30F. Anybody trying to replicate locally will need to know how to recreate. Better would probably be adding an arg to pass this path that defaults to this for the runner

[ScottTodd]

General workflow tip for tests / CI: start by making tests runnable by a developer, then teach a CI machine how to follow those steps. We should never have a hardcoded path like this (and this will only work on systems that support such a file path, so no Windows).

[IanNod]

nit: would be good to comment why we have the xfails for any newcomers not familiar with what isn't functional yet

[ScottTodd]

Let's keep an eye on these flags to make sure we're measuring something representative with what codegen is working on and what model serving will do. These flags change behavior fairly significantly.

[saienduri]

Looks like artifacts are not being uploaded as part of the latest run? Where are we dumping to ${{ github.workspace }}/${{ steps.date.outputs.date }}?

[saienduri]

Nice, everything works!