Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Throw error on buffers of length > 16m #84

Open
bmschmidt opened this issue Jun 13, 2023 · 0 comments
Open

Throw error on buffers of length > 16m #84

bmschmidt opened this issue Jun 13, 2023 · 0 comments

Comments

@bmschmidt
Copy link
Collaborator

bmschmidt commented Jun 13, 2023

We don't define a limit on tile size, but there is a hardcoded parameter in MultipurposeBufferSet that assumes no single column will be more than 64MB. That means that tiles of size greater than 16 million or individual arrow record batches greater than that will completely break deepscatter. It's unlikely anyone would do that, but there should be an error which I don't think there is.

Note that the issue here is about individual tiles--more than 16m points in a dataset is completely fine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant