Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Querys that return large # of results can freeze / crash the app #137

Closed
Tracked by #154
matthewmturner opened this issue Sep 12, 2024 · 2 comments · Fixed by #163 or #157 · May be fixed by #141
Closed
Tracked by #154

Querys that return large # of results can freeze / crash the app #137

matthewmturner opened this issue Sep 12, 2024 · 2 comments · Fixed by #163 or #157 · May be fixed by #141

Comments

@matthewmturner
Copy link
Collaborator

I ran a query on local context that returned a large number of rows and the app became unresponsive after completing. We need to paginate somehow

image
@matthewmturner
Copy link
Collaborator Author

@alamb FYI pagination also relevant for local context

@alamb
Copy link
Contributor

alamb commented Sep 12, 2024

@alamb FYI pagination also relevant for local context

I agree

One way to do this might be to wrap the SendableRecordBatchStream in an object that buffers past results and fetches next result on demand. Something like this perhaps

struct SendableRecordBatchStream {
  // currently executing stream
  inner: SendableRecordBatchStream,
  // any batches that have been buffered so far
  batches: Vec<RecordBatch>,
  // current batch being shown
  current_batch: usize,
}

impl SendableRecordBatchStream {
  /// return the next batch in the stream
  async fn next(&self) -> &RecordBatch {
    self.current_batch += 1;
    self.current()
  }

  /// go to current_index
 async fn current(&self) -> &RecordBatch{
    while self.batches.len()+1 < current_batch {
      // fetch next batch from inner and put into self.batches
    }
    self.batches[self.current_batch]
  }
}

@matthewmturner matthewmturner linked a pull request Sep 13, 2024 that will close this issue
2 tasks
@matthewmturner matthewmturner mentioned this issue Sep 20, 2024
20 tasks
This was linked to pull requests Oct 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants