refactor: Zero-Field Structs and DataFrame with Height Property #19123

coastalwhite · 2024-10-07T13:01:35Z

This PR refactors a large part of the code to allow for:

Zero-Field Structs (ZFSs)
Zero-Column DataFrame (with non-zero height) (ZCDFs)

This required quite a bit of changes all over the place to be able to support.

[skip ci]

codecov · 2024-10-08T17:02:40Z

Codecov Report

Attention: Patch coverage is 82.30088% with 120 lines in your changes missing coverage. Please review.

Project coverage is 79.79%. Comparing base (9dada18) to head (d87fbf8).
Report is 40 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-core/src/frame/mod.rs	81.01%	30 Missing ⚠️
crates/polars-arrow/src/array/struct_/mutable.rs	0.00%	19 Missing ⚠️
...tream/src/nodes/parquet_source/row_group_decode.rs	0.00%	17 Missing ⚠️
crates/polars-core/src/serde/series.rs	42.85%	12 Missing ⚠️
crates/polars-core/src/chunked_array/ops/bits.rs	43.75%	9 Missing ⚠️
crates/polars-arrow/src/legacy/array/mod.rs	0.00%	7 Missing ⚠️
crates/polars-arrow/src/array/struct_/mod.rs	87.50%	3 Missing ⚠️
...rates/polars-core/src/chunked_array/struct_/mod.rs	94.23%	3 Missing ⚠️
crates/polars-core/src/frame/row/av_buffer.rs	82.35%	3 Missing ⚠️
...s-pipe/src/executors/sinks/group_by/generic/mod.rs	0.00%	3 Missing ⚠️
... and 12 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #19123      +/-   ##
==========================================
+ Coverage   79.78%   79.79%   +0.01%     
==========================================
  Files        1531     1532       +1     
  Lines      208445   208649     +204     
  Branches     2913     2913              
==========================================
+ Hits       166301   166498     +197     
- Misses      41593    41601       +8     
+ Partials      551      550       -1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

coastalwhite · 2024-10-08T16:40:08Z

crates/polars-arrow/src/mmap/mod.rs

-unsafe fn _mmap_unchecked<T: AsRef<[u8]>>(
-    fields: &ArrowSchema,


drive-by: this function was unused

coastalwhite · 2024-10-08T16:43:27Z

crates/polars-core/src/chunked_array/comparison/mod.rs

        let mut out = a
            .fields_as_series()
            .iter()
            .zip(b.fields_as_series().iter())
            .map(|(l, r)| op(l, r))
            .reduce(reduce)
-            .unwrap();
+            .unwrap_or_else(|| BooleanChunked::full(PlSmallStr::EMPTY, !value, a.len()));


if both structs are ZFSs it is always eq / not ne

coastalwhite · 2024-10-09T06:10:59Z

crates/polars-core/src/chunked_array/random.rs

@@ -192,10 +192,7 @@ impl DataFrame {

        match n.get(0) {
            Some(n) => self.sample_n_literal(n as usize, with_replacement, shuffle, seed),
-            None => {


drive-by: this should be the same

coastalwhite · 2024-10-09T06:11:09Z

crates/polars-core/src/chunked_array/random.rs

@@ -237,10 +234,7 @@ impl DataFrame {
                let n = (self.height() as f64 * frac) as usize;
                self.sample_n_literal(n, with_replacement, shuffle, seed)
            },
-            None => {


drive-by: this should be the same

coastalwhite · 2024-10-09T06:23:41Z

crates/polars-core/src/frame/horizontal.rs

@@ -32,6 +32,15 @@ impl DataFrame {
    /// - the length of all [`Column`] is equal to the height of this [`DataFrame`]
    /// - the columns names are unique
    pub unsafe fn hstack_mut_unchecked(&mut self, columns: &[Column]) -> &mut Self {
+        // If we don't have any columns yet, copy the height from the given columns.
+        if let Some(fst) = columns.first() {
+            if self.width() == 0 {


maybe this should have a self.height() == 0 check as well, but I want to leave that as future correctness work

coastalwhite · 2024-10-09T06:31:31Z

crates/polars-core/src/frame/mod.rs

@@ -1232,6 +1292,10 @@ impl DataFrame {
        if let Some(idx) = self.get_column_index(column.name().as_str()) {
            self.replace_column(idx, column)?;
        } else {
+            if self.width() == 0 {


coastalwhite · 2024-10-09T06:31:55Z

crates/polars-core/src/frame/mod.rs

@@ -1274,7 +1338,13 @@ impl DataFrame {
        debug_assert!(self.width() == 0 || self.height() == column.len());
        debug_assert!(self.get_column_index(column.name().as_str()).is_none());

+        // SAFETY: Invariant of function guarantees for case `width` > 0. We set the height
+        // properly for `width` == 0.
+        if self.width() == 0 {


coastalwhite · 2024-10-09T06:32:07Z

crates/polars-core/src/frame/mod.rs

@@ -1288,6 +1358,10 @@ impl DataFrame {
                self.replace_column(idx, c)?;
            }
        } else {
+            if self.width() == 0 {


coastalwhite · 2024-10-09T06:36:39Z

crates/polars-core/src/series/any_value.rs

@@ -657,13 +657,6 @@ fn any_values_to_list(
                    DataType::Categorical(Some(Arc::new(RevMapping::default())), *ordering)
                },

-                // Structs don't support empty fields yet.


It is very nice that we can remove these here.

coastalwhite · 2024-10-09T06:42:54Z

crates/polars-io/src/hive.rs

            .iter()
-            .map(|s| s.new_from_index(0, num_rows).into());


drive-by: just convert into scalar column

utkarshgupta137 · 2024-10-27T17:56:08Z

crates/polars-arrow/src/array/struct_/mutable.rs

-    }
-
-    /// The mutable values
-    pub fn mut_values(&mut self) -> &mut Vec<Box<dyn MutableArray>> {


It feels like this method was removed by accident. I'm not a polars expert, but I don't think there is a way to mutate the data without mut_values() or value().

It was not removed by accident. Exposing this data and mutating this data may break invariants of the MutableStructArray. For example, you may change the length of the columns which would make the freeze output invalid. At the moment, the only way to mutate items is to append to the MutableStructArray.

Sorry, I'm not able to figure out how to mutate data anymore. Could you please share an example?

github-actions bot added internal An internal refactor or improvement python Related to Python Polars rust Related to Rust Polars labels Oct 7, 2024

coastalwhite added 6 commits October 8, 2024 14:39

big boy commit

0a783c2

[skip ci]

all tests working still some points to clean up

118a25e

[skip ci]

add serialization

7261a5b

add tests, fix ipc, avro and arrow

cf5c63b

fix avro and growable

3beca20

last changes

11dbc10

coastalwhite force-pushed the refactor/zero-field-struct branch from 2445782 to 11dbc10 Compare October 8, 2024 12:39

coastalwhite added 9 commits October 8, 2024 14:43

add missing file

b90b5bb

fix typo

fcd7bce

remove failing test

c9abf96

fix slice

1d41bde

rustfmt

520a6b4

remove dbg oops

b1a8d9b

fix doc test

7a7440f

update MutableStructArray error message

c692565

remove unnecessary to_string

a9b9303

coastalwhite added 2 commits October 9, 2024 08:17

add better SAFETY comment for empty struct construction

8f65209

add comment on why we are clearing

d87fbf8

coastalwhite commented Oct 9, 2024

View reviewed changes

coastalwhite marked this pull request as ready for review October 9, 2024 07:36

coastalwhite requested review from ritchie46, c-peters, alexander-beedie, MarcoGorelli, reswqa and orlp as code owners October 9, 2024 07:36

ritchie46 approved these changes Oct 11, 2024

View reviewed changes

ritchie46 merged commit dbbd93f into pola-rs:main Oct 11, 2024
25 checks passed

coastalwhite deleted the refactor/zero-field-struct branch October 11, 2024 07:12

ritchie46 mentioned this pull request Oct 11, 2024

Empty struct's value change since version 1.3.0 when calling to_dicts #19147

Closed

2 tasks

c-peters added the accepted Ready for implementation label Oct 14, 2024

c-peters assigned coastalwhite Oct 14, 2024

utkarshgupta137 reviewed Oct 27, 2024

View reviewed changes

utkarshgupta137 mentioned this pull request Nov 1, 2024

[rust-0.44] No way to mutate MutableStructArray #19585

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: Zero-Field Structs and DataFrame with Height Property #19123

refactor: Zero-Field Structs and DataFrame with Height Property #19123

coastalwhite commented Oct 7, 2024 •

edited

Loading

codecov bot commented Oct 8, 2024 •

edited

Loading

coastalwhite Oct 8, 2024

coastalwhite Oct 8, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

coastalwhite Oct 9, 2024

utkarshgupta137 Oct 27, 2024

coastalwhite Oct 28, 2024

utkarshgupta137 Oct 28, 2024 •

edited

Loading

		unsafe fn _mmap_unchecked<T: AsRef<[u8]>>(
		fields: &ArrowSchema,

refactor: Zero-Field Structs and DataFrame with Height Property #19123

refactor: Zero-Field Structs and DataFrame with Height Property #19123

Conversation

coastalwhite commented Oct 7, 2024 • edited Loading

codecov bot commented Oct 8, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

utkarshgupta137 Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

coastalwhite commented Oct 7, 2024 •

edited

Loading

codecov bot commented Oct 8, 2024 •

edited

Loading

utkarshgupta137 Oct 28, 2024 •

edited

Loading