Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vectorized file writes #20

Merged
merged 13 commits into from
Dec 6, 2024
Merged

Vectorized file writes #20

merged 13 commits into from
Dec 6, 2024

Conversation

aliddell
Copy link
Member

@aliddell aliddell commented Nov 27, 2024

Adds a class which takes a std::vector<std::span<std::byte>>>& and writes the contents of each span in a vectorized fashion to the filesystem. Includes some system-dependent code because of course it does.

Also adds benchmarking code so that we can test streaming performance before and after this change.

@aliddell aliddell marked this pull request as ready for review December 2, 2024 14:43
@aliddell aliddell requested a review from jeskesen December 2, 2024 14:43
Copy link
Contributor

@jeskesen jeskesen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there's a lot of duplicated code between ChunkIndexBuffer & ShardIndexBuffer. Could they be combined?

@aliddell
Copy link
Member Author

aliddell commented Dec 5, 2024

It looks like there's a lot of duplicated code between ChunkIndexBuffer & ShardIndexBuffer. Could they be combined?

Actually they don't belong there. Those are artifacts from something I was trying but discarded.

@aliddell aliddell requested a review from jeskesen December 5, 2024 18:22
Copy link
Contributor

@jeskesen jeskesen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aliddell aliddell merged commit 5f613e8 into main Dec 6, 2024
7 checks passed
@aliddell aliddell deleted the vector-writes branch December 6, 2024 18:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants