Add e2e parameterization using file

[erman-gurses]

This PR and future related PRs will be transformed into performance-CI, benchmarking different operators with various input parameters. For now, we are starting with GEMMs. In the future, we will add convolution, flash attention, and other operators as needed. The ultimate goal is to create performance plots for those operators so we can see their performance daily.

The perf.yaml file is added to the repo to distinguish between Performance and CI related operations.

[Hardcode84]

LGTM

[ScottTodd]

(Saw this after a comment on #163 (comment))

Did you consider using pytest marks instead of environment variables and json files?

If all you want to enable is running different sets of tests in different workflows, pytest marks may be a better tool for the job than this configuration coming from files and environment variables.

• https://docs.pytest.org/en/stable/how-to/mark.html
• https://docs.pytest.org/en/stable/example/markers.html#mark-examples

With marks, you can define all the tests in the source file and then developers and automated workflows can choose to skip tests with a given mark (e.g. "large", "large_shapes", "benchmark", etc.).

This environment variable approach gives you more flexibility to customize the shapes without modifying the test source, which can help when running the tests from outside of this repository, but it doesn't look like that is a case you want to support at this moment. I think the ergonomics of setting an environment variable to a specific file path are quite poor for local development, especially if you plan on adding more of these files (one per test? each with its own environment variable?)