tests: add qrexec performance tests #647
Conversation
|
I think it would be useful to provide a standalone script that users can run on their own machines, without having to stop qubesd or do something similarly disruptive. The integration tests would then call the script with the appropriate arguments. |
|
And what that script would do? Just call call qrexec in a loop? That's oneliner you can simply copy from the test to shell. OTOH, if you'd expect it to create VMs, setup policy etc separately, that would duplicate several parts that our test classes already do (and probably have different bugs while at it). And BTW, creating/removing VMs, adjusting policy etc can be considered "disruptive" too, so it wouldn't even solve your issue. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #647 +/- ##
==========================================
- Coverage 69.78% 69.71% -0.07%
==========================================
Files 58 58
Lines 12499 12499
==========================================
- Hits 8722 8714 -8
- Misses 3777 3785 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
OpenQA test summaryComplete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2025012615-4.3-archlinux&flavor=templates Test run included the following:
New failures, excluding unstableCompared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024111705-4.3&flavor=update
Failed tests11 failures
Fixed failuresCompared to: https://openqa.qubes-os.org/tests/119126#dependencies 1 fixed
Unstable tests |
|
Thinking more about it, maybe I can make a script that takes already created (and running?) VMs as an argument. |
I think this is a good idea. While users could of course make their own benchmarks, having a single benchmark script is much better for reproducibility. |
|
So, the script is there. It needed a bit of boilerplate compared to just running test commands directly, but well... |
Add simple connection latency, and throughput tests. Run them with different type of services (scripts, socket, via fork-server or not). They print a test run time for comparison - the lower the better. The tests can be also started outside of the full test run by calling /usr/lib/qubes/tests/qrexec_perf.py. It requires giving names of two existing and running VMs. QubesOS/qubes-issues#5740
Add simple connection latency, and throughput tests. Run them with
different type of services (scripts, socket, via fork-server or not).
They print a test run time for comparison - the lower the better.
QubesOS/qubes-issues#5740
This will be especially useful to measure impact of QubesOS/qubes-core-qrexec#141 and similar changes.