Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ref: move prepared files to ~/.cache dir #781

Merged
merged 9 commits into from
Oct 5, 2023

Conversation

jordanrfrazier
Copy link
Collaborator

@jordanrfrazier jordanrfrazier commented Sep 28, 2023

Moves the prepared files directory to ~/.cache/kaskada/.

The other alternative is to, for now, create a temporary directory that cleans up after use. While the ~/.cache is generally used for application-specific files (which is what the prepared files are), we aren't utilizing them and won't until we have state. But, if it's not a large concern to accumulate prepared files in the cache, we might as well start with it here in anticipation of using them.

Closes #779

@cla-bot cla-bot bot added the cla-signed Set when all authors of a PR have signed our CLA label Sep 28, 2023
@bjchambers
Copy link
Collaborator

Until we start re-using them, we really should delete them. Otherwise, every run of the test creates files that build up over time, eventually leading to out of disk space. And now it's in a hidden cache directory.

I think:

  1. Preparing to a temporary or cache directory and deleting it is the way to go for now.
  2. Soon, we want to re-enable prepare to S3 (more storage) and then (possibly) use the local cache directory for things like the rocksdb and a cache of downloaded files... at which point we should manage it like a cache and limit the size, etc.

crates/sparrow-runtime/src/prepare/preparer.rs Outdated Show resolved Hide resolved
crates/sparrow-runtime/src/stores/object_store_url.rs Outdated Show resolved Hide resolved
crates/sparrow-runtime/src/prepare/preparer.rs Outdated Show resolved Hide resolved
crates/sparrow-session/src/session.rs Outdated Show resolved Hide resolved
crates/sparrow-session/src/table.rs Show resolved Hide resolved
@jordanrfrazier jordanrfrazier added this pull request to the merge queue Oct 5, 2023
Merged via the queue into main with commit 147642b Oct 5, 2023
32 checks passed
@jordanrfrazier jordanrfrazier deleted the move-prepared-data-directory branch October 5, 2023 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla-signed Set when all authors of a PR have signed our CLA refactor sparrow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: Tests running prepare leave the tables directory
2 participants