bench | locli: new DB persistence backend for analysis data #6054
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR adds a new database persistence backend to the benchmark analysis tool
locli
.Persisting log objects in a serverless SQLite DB has proven several advantages over the current file based solution (FS). Considering a perf cluster run with abundant raw log data:
Furthermore, the DB backend opens up a way to carry out custom quick queries into the analysis input - either on the application level or in SQL. A suitable abstraction for those will be a future endeavor; this PR adds a new testbed application
locli-quick
for development (which, for that reason, needs not be aworkbench
shell input as of now).Last not least, default RTS GC settings for
locli
have been revisited. This PR changes the default such that on the perf cluster deployer, a standard analysis (almost) fits into 64GiB, and as a result there's (almost) no swapping going on. This is translates into a significant speed improvement, regardless of which persistence backend is used.The PR implements all changes listed in
locli
's changelog, bumping it to a new major version2.0
:prepare-db
andunlog-db
to create and read from DB persistence backend respectivelyLogObjectSource
to represent input from different backends (file or DB)locli-quick
which aims to be a development testbed for (upcoming) DB-backed quick queriesIt also integrates the new
locli
feature intoworkbench
such that:$WB_LOCLI_DB
is introduced, defaulting to0
--locli-db
1
results in using the DB for both preparation and analysis stepsThis means the changes of this PR are currently "opt-in". All existing functionality is retained, unmodified, and as such, this PR is non-breaking wrt. to existing workflows.
Checklist
See Runnings tests for more details
CHANGELOG.md
for affected package.cabal
files are updatedhlint
. See.github/workflows/check-hlint.yml
to get thehlint
versionstylish-haskell
. See.github/workflows/stylish-haskell.yml
to get thestylish-haskell
versionghc-8.10.7
andghc-9.2.7