Rewrite existing benchmarks to improve cross-platform support and cover additional tools #90
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Updates
download.sh
was replaced with the moduledata_source.rs
. In theory this should work on any platform. Instead of storing files to/tmp
, files are stored withintarget/test_data
. This has the benefit of working in non-unix environments (windows) and means test data will be removed upon runningcargo clean
.cargo bench
. MD5 checksums (calculated via themd5
crate) are used to check if current test data is up-to-date. If a file is not present or the checksum does not match then the file is downloaded. This should not effect the perceived time to run benchmarks after the initial download and has the added bonus of ensuring that anyone running the benchmarks has the correct version of the test data.which
crate). The programs that are found and their respective paths are printed before the benchmark begins. Benchmarks are grouped by the data file being read so they will be compared on the same page within criterion's generated reports.bgpkit-parser
against other programs, thebgpkit-parser
executable is used instead of directly calling into the library. Despite the increased overhead of calling a separate executable,bgpkit-parser
has still managed to beatbgpdump
in every benchmark I have done. Using the executable is possible because Cargo treats benchmarks and tests placed within thebenches
andtests
folders separate from thesrc
directory as integration tests. To support integration tests/benchmarks, all binary targets are built prior to starting. In the case of benches, binary targets are built in release mode.Additional Notes
bgpkit-parser
andbgpdump
. In theory it should work automatically when those additional tools are installed to the system, but it may need some tweaks for the exact command line arguments.rib-example.bz2
?bgpdump
takes an extremely long time to go through this file. By criterion's estimate, it would take a full hour on my machine to collect the minimum 10 samples required and I did not want to wait that long.