test: add a small set of ops.testing benchmark tests #1504
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tonyandrewmeyer
changed the title
benhoyt
reviewed
Dec 18, 2024
Also more cleanly separate out ops and ops-scenario, to be consistent with what we do elsewhere. There are no ops benchmark tests yet but put the bits in place to add them, since we will want to have some at some point.
…oject.toml Ideally, we would statically check these, like the other tests, but it does not seem super important for benchmarking tests. The issue is that there is an import after a sys.path manipulation, and pyright cannot figure that out, so complains about a lot of things. We do not want to have that charm on the path, and adjusting the path for pyright does not seem great either. Not clear what the cleanest solution is.
benhoyt
approved these changes
Dec 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The PR:
tox -e benchmarkcommand.test/test_benchmark.pymodel.tox -e unitandtox -e coverageI wrote a small set of benchmark tests while working on #1434. It seems like it'd be worth keeping these - in the future, they could be expanded to include a small set of benchmark tests that target ops more specifically, and ideally we could have a CI workflow that failed if there were regressions (currently, running the tests just outputs the timing, and you can use the pytest-benchmark tools to do comparisons).