Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bench: add repro test #10167

Closed
wants to merge 2 commits into from
Closed

bench: add repro test #10167

wants to merge 2 commits into from

Conversation

dberenbaum
Copy link
Collaborator

@dberenbaum dberenbaum commented Dec 14, 2023

Closes iterative/dvc-bench#468

Edit: the need for this is to help track performance for issues like #9085

@dberenbaum dberenbaum requested a review from a team December 14, 2023 19:21
Copy link

codecov bot commented Dec 14, 2023

Codecov Report

Attention: 8 lines in your changes are missing coverage. Please review.

Comparison is base (f3fa674) 90.62% compared to head (d9abbe7) 90.60%.
Report is 2 commits behind head on main.

Files Patch % Lines
dvc/testing/benchmarks/cli/commands/test_repro.py 0.00% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10167      +/-   ##
==========================================
- Coverage   90.62%   90.60%   -0.02%     
==========================================
  Files         500      501       +1     
  Lines       37852    37879      +27     
  Branches     5503     5508       +5     
==========================================
+ Hits        34305    34322      +17     
- Misses       2908     2917       +9     
- Partials      639      640       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@skshetry
Copy link
Member

It's going to be hard to write a fair benchmark for dvc repro.

I understand the motivation, but what you are testing here is dvc commit and that too is a no-op here because all of those were run already with run_copy() and is cached (mtimes, md5 hashes and the actual files).

Even if you made all of them --no-exec and tried to benchmark dvc repro, there is I/O happening in the subprocess cmd, which will be a significant part of the runtime.

@dberenbaum dberenbaum marked this pull request as draft December 19, 2023 13:51
@dberenbaum dberenbaum closed this Feb 29, 2024
@skshetry skshetry deleted the bench-repro branch March 5, 2024 08:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add pipelines use case
3 participants