Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return array of buffers instead of buffer.concat #54

Open
ZJONSSON opened this issue Feb 16, 2018 · 1 comment
Open

Return array of buffers instead of buffer.concat #54

ZJONSSON opened this issue Feb 16, 2018 · 1 comment

Comments

@ZJONSSON
Copy link
Contributor

Buffer.concat, particularly on the whole rowgroup buffers can get expensive memory wise, as we'll end up with two copies of the whole buffer (the individual parts and the concatenated version) for a moment before the garbage-collector picks up the pieces. This can limit the maximum size of the rowgroup.

It might be more efficient to return bodyParts (a sequential array of bufffers) in general instead of buffer.concat and then in writeSection we loop through the bodyParts and write them sequentially.

@ZJONSSON ZJONSSON changed the title Replace returning buffer.concat with an array of buffers Return array of buffers instead of buffer.concat Feb 16, 2018
@ZJONSSON
Copy link
Contributor Author

see https://github.com/ZJONSSON/parquetjs/tree/bodyParts for WIP (contingent on #52)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant