-
Notifications
You must be signed in to change notification settings - Fork 5.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Benchmarking external tests #12441
Benchmarking external tests #12441
Conversation
b1ce656
to
37f2c54
Compare
9bc1272
to
2258a08
Compare
37f2c54
to
e22f0cf
Compare
7a5c835
to
20c256d
Compare
e22f0cf
to
c40a44f
Compare
9280182
to
7864565
Compare
This is pretty much done now. Benchmark results for Gnosis are still incomplete, probably because its tests are not based on mocha but they will work after the switch to Hardhat in #12195 so there's no point fixing that here. For Colony I'm not sure if they're complete but it only runs on nightly so I'm leaving it as is for now. We should update it to the latest version first anyway. When reviewing check the artifacts of the |
c40a44f
to
1928b78
Compare
7864565
to
39b6206
Compare
39b6206
to
4806603
Compare
14ed1af
to
4c76add
Compare
Rebased on the Uniswap PR (#12532) since that's likely to get merged soon and that way I can already include it here.. |
fad1902
to
7b943d4
Compare
1a6c62e
to
aeb9637
Compare
f765830
to
0ee4835
Compare
0ee4835
to
741404b
Compare
741404b
to
55a7ca5
Compare
d9db164
to
77afad2
Compare
I updated the code for new external tests. This should now pass CI. I also fixed an issue with benchmark results being missing for PRBMath and Bleeps. It looks like running tests via Unfortunately results seem incomplete in both cases. In Bleeps the table has no deployment costs while in PRBMath method call costs are missing. I'm not sure why so I asked for help in PaulRBerg/prb-math#70. In any case, I think we should merge it even in the current state. Benchmark results are just extra information and jobs are on purpose designed in such a way that not having them does not make them fail. |
Regarding no calls in PRBMath - mystery solved. All the methods called in tests are There are plans to make hardhat-gas-reporter inject |
As for Bleeps, the issue is caused by the use of hardhat-deploy plugin. It deploys the contracts before tests are executed and hardhat-gas-reporter does not monitor these calls. The project actually has a separate command to report deployment gas. There's an issue to fix that in hardhat-gas-reporter though: cgewecke/hardhat-gas-reporter#86. So this is another thing that should just work once it's fixed upstream. |
77afad2
to
26afbf3
Compare
26afbf3
to
60d9aa0
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I vote for just merging this and seeing how it performs.
I by far have not done a complete line-by-line review, but since
(1) all of this is purely informative and non-critical, yet also (2) the information
will be quite helpful and useful, I think it may be reasonable to merge
and see how well it works without a full rigorous review.
Depends on #12440.Merged.Depends on #12532.Merged.Depends on #12629.Merged.This PR adds a mechanism for gathering a set of metrics from all external tests and combining them into an easily diffable JSON report.
Currently it gathers the following information:
jq
.The biggest part (report creation) is already implemented but there are still a few small things to iron out:
Colony(only runs on nigthly)Gnosis Safe(will work after Run GnosisSafe external tests with Hardhat and directly on upstream #12195)