Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing test_bench configuration support #2590

Open
shink opened this issue Feb 11, 2025 · 1 comment · May be fixed by #2592
Open

Missing test_bench configuration support #2590

shink opened this issue Feb 11, 2025 · 1 comment · May be fixed by #2592

Comments

@shink
Copy link
Contributor

shink commented Feb 11, 2025

Seems --config is defined but not used actually in the test_bench userbenchmark. I would like to add support for it if this change makes sense.

parser.add_argument(
"--config", "-c", default=None, help="YAML config to specify tests to run."
)

cc: @xuzhao9

@shink
Copy link
Contributor Author

shink commented Feb 12, 2025

Actually, I'm working on a PyTorch out-of-tree test infra project, which is using torchbenchmark to test out-of-tree devices and can be found at here.
And we want to let each device vendor define their own configuration, might like this:

devices:
  - "foo"
models:
  - model: BERT_pytorch
    batch_size: 1

  - model: yolov3
    skip: true
extra_args:
  - "--accuracy"
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant