Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite existing benchmarks to improve cross-platform support and cover additional tools #90

Merged
merged 3 commits into from
Feb 25, 2023

Conversation

jmeggitt
Copy link
Contributor

Updates

  • download.sh was replaced with the module data_source.rs. In theory this should work on any platform. Instead of storing files to /tmp, files are stored within target/test_data. This has the benefit of working in non-unix environments (windows) and means test data will be removed upon running cargo clean.
  • Test data is downloaded as part of running cargo bench. MD5 checksums (calculated via the md5 crate) are used to check if current test data is up-to-date. If a file is not present or the checksum does not match then the file is downloaded. This should not effect the perceived time to run benchmarks after the initial download and has the added bonus of ensuring that anyone running the benchmarks has the correct version of the test data.
  • As part of implementing [feature] add benchmark #12, the benchmark will attempt to search for each BGP parsing program on the system (using the which crate). The programs that are found and their respective paths are printed before the benchmark begins. Benchmarks are grouped by the data file being read so they will be compared on the same page within criterion's generated reports.
  • To more fairly compare bgpkit-parser against other programs, the bgpkit-parser executable is used instead of directly calling into the library. Despite the increased overhead of calling a separate executable, bgpkit-parser has still managed to beat bgpdump in every benchmark I have done. Using the executable is possible because Cargo treats benchmarks and tests placed within the benches and tests folders separate from the src directory as integration tests. To support integration tests/benchmarks, all binary targets are built prior to starting. In the case of benches, binary targets are built in release mode.

Additional Notes

  • While I included all of the programs listed under [feature] add benchmark #12, I only tested it with bgpkit-parser and bgpdump. In theory it should work automatically when those additional tools are installed to the system, but it may need some tweaks for the exact command line arguments.
  • Are there any smaller table dump test files available than rib-example.bz2? bgpdump takes an extremely long time to go through this file. By criterion's estimate, it would take a full hour on my machine to collect the minimum 10 samples required and I did not want to wait that long.

@digizeph digizeph self-assigned this Feb 24, 2023
@digizeph digizeph added this to the V0.9 milestone Feb 24, 2023
@digizeph digizeph merged commit 2dfdd96 into bgpkit:main Feb 25, 2023
@digizeph
Copy link
Member

@jmeggitt I've merged this PR and added a smaller RIB dump file (2.7MB) in c51813f. Thanks for the efforts!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants