Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RFC] Benchmarking Requirements #101

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[RFC] Benchmarking Requirements #101

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions rfcs/20231212-benchmarking-rfc.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# OpenXLA Benchmarking
| Status | Proposed |
| :------------ | :------------------------------------------------------ |
| **RFC #** | [101](https://github.com/openxla/community/pull/101) |
| **Authors** | David Dunleavy ([email protected]) |
| **Sponsor** | Puneith Kaul ([email protected]) |
| **Updated** | 2023-12-12

## Objective
Gather feedback on what benchmarking infrastructure would be helpful for the OpenXLA community.

## Proposal
Previous benchmarking efforts like those described in the
[Benchmarking Strategy RFC](https://github.com/openxla/community/blob/main/rfcs/20230505-benchmarking-strategy.md)
have been disrupted due to changes in broader strategy in the OpenXLA ecosystem.
Since then, there have been some independent discussions about benchmarking for XLA,
but we are interested in getting a pulse on what's most important to the community.

## Questions
What openly available models are most interesting to measure?
Is using models in HLO/StableHLO form sufficient? Or is it preferable to do framework level benchmarks?
What backends/targets/configurations are most important to benchmark? Single vs multi-host?