MapArray Additional Nullability

[tustvold]

Which part is this question about

A MapArray is represented in parquet as follows, see here.

<map-repetition> group <name> (MAP) {
  repeated group key_value {
    required <key-type> key;
    <value-repetition> <value-type> value;
  }
}

However, it is represented in arrow as

Field::new(
    <name>,
    DataType::Map(
        Box::new(Field::new("entries", DataType::Struct(..), <nullable>))
        <is_sorted>
    )
   <nullable>
)

The confusion lies in the fact we now have nullability at three levels:

• Within the MapArray
• Within the StructArray
• Within the key and value arrays

When the original parquet data only has nullability at two levels.

The writer logic appears to ignore the nullability of the inner StructArray, which doesn't feel right.

Describe your question

• Should the inner StructArray always be not nullable?
• Could we store DataType instead of Field in DataType::Map
• Could we store the child DataType directly in DataType::Map

Additional context

#1642 also relates to the semantics of MapArrays

Map Arrays were added by @nevi-me as part of #395

[wjones127]

FYI, the arrow spec says:

Neither the "entries" field nor the "key" field may be nullable.

So it's safe to ignore the nullability at the struct level and for the key field.

[ByteBaker]

@alamb can you please review this once to confirm if this still has anything remaining?