Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IGUANA cannot handle queries larger than ~2GB #176

Closed
liss-h opened this issue Sep 21, 2022 · 2 comments · Fixed by #266
Closed

IGUANA cannot handle queries larger than ~2GB #176

liss-h opened this issue Sep 21, 2022 · 2 comments · Fixed by #266
Assignees
@nck-mlcnv
Labels
bug

Comments

@liss-h
Copy link

liss-h commented Sep 21, 2022

Java arrays and probably some other things are for some reason limited to roughtly Integer.MAX_VALUE (= 2^32 / 2) bytes which is just over 2GB. Which is problematic when the query file is larger than that (which can happen in DELETE DATA or INSERT DATA queries) as IGUANA will just crash with an out of memory error.

I don't know if that is fixable
@liss-h liss-h added the bug label Sep 21, 2022
@liss-h liss-h changed the title IGUANA cannot handle queries larger than 2^32 / 2 bytes IGUANA cannot handle queries larger than ~2GB bytes Sep 21, 2022
@liss-h liss-h changed the title IGUANA cannot handle queries larger than ~2GB bytes IGUANA cannot handle queries larger than ~2GB Sep 21, 2022
@bigerl
Copy link
Member

bigerl commented Mar 8, 2023

  • Larger queries should always be streamed from disk and not reside at RAM. Configurable threshold.
  • Supporting larger queries requires Array with long size type (or streams?)
  • two modes:
    • chunked HTTP without content-length
    • regular HTTP request

@bigerl
Copy link
Member

bigerl commented Aug 7, 2024

This is almost solved. The only remaining issue is:

  • Right now, we can only hash queries up to 2GB because of the hashing scheme used. We should use a streaming XXH3 instead

@nck-mlcnv nck-mlcnv linked a pull request Aug 7, 2024 that will close this issue
Fix issues with queries that are bigger than 2GB #266
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nck-mlcnv nck-mlcnv

Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix issues with queries that are bigger than 2GB dice-group/IGUANA
3 participants
@bigerl @liss-h @nck-mlcnv