Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: parts of hottier #1022

Merged
merged 38 commits into from
Jan 28, 2025
Merged

refactor: parts of hottier #1022

merged 38 commits into from
Jan 28, 2025

Conversation

de-sh
Copy link
Contributor

@de-sh de-sh commented Dec 5, 2024

Fixes #XXXX.

Description

  • Uses FuturesUnordered to manage async tasks
  • Reuses get_stream_path_for_date where possible
  • Refactors human_size parsing
  • Streams through results instead of collecting
  • Stores sizes as integers in memory

This PR has:

  • been tested to ensure log ingestion and log query works.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added documentation for new or modified features or behaviors.

@coveralls
Copy link

coveralls commented Dec 17, 2024

Pull Request Test Coverage Report for Build 13006222056

Details

  • 34 of 250 (13.6%) changed or added relevant lines in 6 files are covered.
  • 10 unchanged lines in 4 files lost coverage.
  • Overall coverage increased (+0.2%) to 12.951%

Changes Missing Coverage Covered Lines Changed/Added Lines %
src/metadata.rs 0 1 0.0%
src/storage/mod.rs 0 1 0.0%
src/validator.rs 0 8 0.0%
src/handlers/http/logstream.rs 0 24 0.0%
src/utils/human_size.rs 34 63 53.97%
src/hottier.rs 0 153 0.0%
Files with Coverage Reduction New Missed Lines %
src/migration/mod.rs 1 0.0%
src/validator.rs 1 0.0%
src/handlers/http/logstream.rs 1 1.77%
src/hottier.rs 7 0.0%
Totals Coverage Status
Change from base Build 13006184900: 0.2%
Covered Lines: 2477
Relevant Lines: 19126

💛 - Coveralls

@de-sh de-sh marked this pull request as ready for review January 14, 2025 08:36
@nikhilsinhaparseable
Copy link
Contributor

@de-sh there is a change in behaviour of the PUT /hottier API
request body expected -
before change: {"size":"10737418240"}
after change: {"size":"10 GiB"}

even the GET /hottier API is failing with error -
Serde(Error("Could not parse given size", line: 1, column: 42))

can you please verify that there should be no behavioural change in the existing APIs.

Copy link
Contributor

@nikhilsinhaparseable nikhilsinhaparseable left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good

@de-sh de-sh changed the title feat: hottier perf work refactor: parts of hottier Jan 27, 2025
@nitisht nitisht merged commit a0a0ec3 into parseablehq:main Jan 28, 2025
11 checks passed
@de-sh de-sh deleted the hottier-perf branch January 28, 2025 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants