Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Maintain unique start_timestamps for indexing #16

Open
desmondcheongzx opened this issue Apr 8, 2021 · 2 comments
Open

Maintain unique start_timestamps for indexing #16

desmondcheongzx opened this issue Apr 8, 2021 · 2 comments

Comments

@desmondcheongzx
Copy link
Collaborator

desmondcheongzx commented Apr 8, 2021

In #14 we introduce a B-tree block index mapping from start_timestamp to filename. There's currently nothing enforcing unique start_timestamps besides an expectation of well-behaved data. We can consider creating a sort of "timestamp checkpoint", where we refuse to keep any data with timestamps less than or equal to our latest start_timestamp.

If we want to avoid losing late data, we can consider keeping the block around for awhile in-between resetting the active block and flushing the old block to disk. Something like Timon.

@n-young
Copy link
Owner

n-young commented Apr 14, 2021

We now map from start_timestamp => vec, I think this fixes it? Close it if you think so

@n-young
Copy link
Owner

n-young commented Apr 15, 2021

New issue: timestamps don't even come remotely in order

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants