Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement serializing to duckdb #1578

Draft
wants to merge 7 commits into
base: main
Choose a base branch
from
Draft

Implement serializing to duckdb #1578

wants to merge 7 commits into from

Conversation

rien
Copy link
Member

@rien rien commented Aug 2, 2024 •
edited
Loading

This PR allows serializing a Dolos analysis to a DuckDB file instead of CSV-files.

Performance Benchmark

Dataset DuckDB write-out DuckDB parse CSV write-out CSV parse
Pyramidal Constants - Exercise 942 ms 4729 ms 622 ms 3202 ms
Pyramidal Constants - Evaluation 481 ms 4654 ms 106 ms 2378 ms
Plutokiller 21 259 ms 11 187 ms 17 557 ms 16 078 ms

Currently only parsing large datasets using DuckDB is faster.

@rien rien added the experiment This is an experiment and might not be released 🧪 label Sep 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
experiment This is an experiment and might not be released 🧪
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@rien