feat: support using general compression for numeric array #3020
Conversation
niyue
changed the title
| compressed_bit_width as usize, | ||
| data_type.clone(), | ||
| ))); | ||
| } else if let Some(compression) = field_meta.and_then(Self::get_field_compression) { |
There was a problem hiding this comment.
When bitpacking is not applicable, check if field itself specifies some compression
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3020 +/- ##
==========================================
+ Coverage 78.19% 78.24% +0.04%
==========================================
Files 239 239
Lines 76782 76870 +88
Branches 76782 76870 +88
==========================================
+ Hits 60043 60148 +105
+ Misses 13669 13633 -36
- Partials 3070 3089 +19
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| let mut encoder: Option<Box<dyn ArrayEncoder>> = None; | ||
|
|
||
| if version >= LanceFileVersion::V2_1 { | ||
| if arrays[0].data_type() == data_type { |
There was a problem hiding this comment.
I definitely think we should bitpack string offsets
If this is desirable, maybe the simplest change is to remove the condition if arrays[0].data_type() == data_type here so that bit packing would be applied. But this condition seems to be added prevously intentionally and I am not sure if this should be simply removed or not.
This PR aims to address issue #3019.
In this PR, a general compression encoder can now be selected for numeric arrays. I’ve reused the existing
CompressedBufferEncoderstruct from the codebase, which currently does not appear to be in use elsewhere.