You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have not yet begun on the next iteration of the vParquet format, but I would like to start collecting ideas for what we would include. The past two iterations have been focused on adding columns and features and so I propose we focus on removing columns and cleanup.
I would like to:
Remove "well known" columns and rely instead on dedicated columns
This would remove a significant amount of complexity in our read/write paths as well reduce footer size
Remove the ServiceStats columns in favor of a marshalled proto or json representation. We only return these on search and don't need them broken out into individual columns.
Drop .list.element from repeated fields column names. We added this for compatibility with some ?? tooling, but we just broke compatibility with other tooling. Personally I'd prefer the simpler names.
Dictionary-less dedicated columns as a place to put fields like json blobs and sql queries. Maybe split the 10 we have into 5 dict and 5 no dict?
Everything is up for discussion. Please put ideas here!
We have not yet begun on the next iteration of the vParquet format, but I would like to start collecting ideas for what we would include. The past two iterations have been focused on adding columns and features and so I propose we focus on removing columns and cleanup.
I would like to:
.list.element
from repeated fields column names. We added this for compatibility with some ?? tooling, but we just broke compatibility with other tooling. Personally I'd prefer the simpler names.Everything is up for discussion. Please put ideas here!
cc @stoewer, @mdisibio, @ndk
The text was updated successfully, but these errors were encountered: