feat(core): Add support for `_file` column #1824

gbrgr · 2025-11-04T09:06:43Z

Which issue does this PR close?

Closes Support for _file metadata column #1766.

What changes are included in this PR?

Integrates virtual field handling for the _file metadata column into RecordBatchTransformer using a pre-computed constants map, eliminating post-processing and duplicate lookups.

Key Changes

New metadata_columns.rs module: Centralized utilities for metadata columns

Constants: RESERVED_FIELD_ID_FILE, RESERVED_COL_NAME_FILE
Helper functions: get_metadata_column_name(), get_metadata_field_id(), is_metadata_field(), is_metadata_column_name()

Enhanced RecordBatchTransformer:

Added constant_fields: HashMap<i32, (DataType, PrimitiveLiteral)> - pre-computed during initialization
New with_constant() method - computes Arrow type once during setup
Updated to use pre-computed types and values (avoids duplicate lookups)
Handles DataType::RunEndEncoded for constant strings (memory efficient)

Simplified reader.rs:

Pass full project_field_ids (including virtual) to RecordBatchTransformer
Single with_constant() call to register _file column
Removed post-processing loop

Updated scan/mod.rs:

Use is_metadata_column_name() and get_metadata_field_id() instead of hardcoded checks

Are these changes tested?

Yes, comprehensive tests have been added to verify the functionality:

New Tests (7 tests added)

Table Scan API Tests (7 tests)

test_select_with_file_column - Verifies basic functionality of selecting _file with regular columns
test_select_file_column_position - Verifies column ordering is preserved
test_select_file_column_only - Tests selecting only the _file column
test_file_column_with_multiple_files - Tests multiple data files scenario
test_file_column_at_start - Tests _file at position 0
test_file_column_at_end - Tests _file at the last position
test_select_with_repeated_column_names - Tests repeated column selection

crates/iceberg/src/scan/mod.rs

vustef · 2025-11-04T11:15:37Z

crates/iceberg/src/scan/mod.rs

+/// // Select regular columns along with the file path
+/// let scan = table
+///     .scan()
+///     .select(["id", "name", RESERVED_COL_NAME_FILE])


How do we ask for _file column without having to explicitly list all the other columns? E.g. get me all columns + _file. There should be some shortcut for this.

crates/iceberg/src/scan/mod.rs

crates/iceberg/src/arrow/reader.rs

…erg-rust into feature/gb/file-column

liurenjie1024

Thanks @gbrgr for this pr. But I think we need to rethink how to compute the _file, _pos metadata column. While it's somehow trivial to compute _file, it's non trivial to compute _pos efficient, since when we read parquet files, we have filtered out some row groups. I think the best way is to push reading these two columns to arrow-rs.

liurenjie1024 · 2025-11-06T10:15:04Z

crates/iceberg/src/arrow/reader.rs

+pub(crate) const RESERVED_FIELD_ID_FILE: i32 = 2147483646;
+
+/// Column name for the file path metadata column per Iceberg spec
+pub(crate) const RESERVED_COL_NAME_FILE: &str = "_file";


I think we should add a metadata_columns module similar to what java did: https://github.com/apache/iceberg/blob/bb32b90c4ad63f037f0bda197cc53fb08c886934/core/src/main/java/org/apache/iceberg/MetadataColumns.java#L2

vustef · 2025-11-06T10:23:35Z

Thanks @gbrgr for this pr. But I think we need to rethink how to compute the _file, _pos metadata column. While it's somehow trivial to compute _file, it's non trivial to compute _pos efficient, since when we read parquet files, we have filtered out some row groups. I think the best way is to push reading these two columns to arrow-rs.

@liurenjie1024 I agree for _pos, and we have a PR there: apache/arrow-rs#8715
But _file seems like something that we don't need the arrow-rs to know about. Similarly, in future, for _row_id from V3 spec, we cannot expect arrow-rs to be responsible for computing that one.

How do we go forward with rethinking this, what would be the action items for us?

…erg-rust into feature/gb/file-column

crates/iceberg/src/arrow/metadata_column_transformer.rs

crates/iceberg/src/arrow/reader.rs

This reverts commit adf0da0.

liurenjie1024

Thanks @gbrgr for this pr, I left some comments to improve.

liurenjie1024 · 2025-11-10T01:15:32Z

crates/iceberg/src/scan/mod.rs

+/// # Ok(())
+/// # }
+/// ```
+pub const RESERVED_COL_NAME_FILE: &str = RESERVED_COL_NAME_FILE_INTERNAL;


We will have more metadata columns, so I would prefert to put these definition in sth like metadata_columns module.

Added a new module

liurenjie1024 · 2025-11-10T01:22:27Z

crates/iceberg/src/scan/mod.rs

        if let Some(column_names) = self.column_names.as_ref() {
            for column_name in column_names {
+                // Skip reserved columns that don't exist in the schema
+                if column_name == RESERVED_COL_NAME_FILE_INTERNAL {


We should have sth like is_metadata_column_name() in metadata_columns module, and useis_metadata_column_name so that we could avoid such changes when we add more metadata columns.

Done in the new module

liurenjie1024 · 2025-11-10T01:32:32Z

crates/iceberg/src/arrow/reader.rs


+    /// Helper function to add a `_file` column to a RecordBatch at a specific position.
+    /// Takes the array, field to add, and position where to insert.
+    fn create_file_field_at_position(


I think this approach is not extensible. I prefer what's similar in this pr:

Add constant_map for ArrowReader

Add another variant of RecordBatchTransformer to handle constant field like _file

I sketched an approach in the record batch transformer, I took the path of the transformer just having a constant_map stored which can be populated by the reader.

liurenjie1024 · 2025-11-10T01:34:46Z

Thanks @gbrgr for this pr. But I think we need to rethink how to compute the _file, _pos metadata column. While it's somehow trivial to compute _file, it's non trivial to compute _pos efficient, since when we read parquet files, we have filtered out some row groups. I think the best way is to push reading these two columns to arrow-rs.

@liurenjie1024 I agree for _pos, and we have a PR there: apache/arrow-rs#8715 But _file seems like something that we don't need the arrow-rs to know about. Similarly, in future, for _row_id from V3 spec, we cannot expect arrow-rs to be responsible for computing that one.

How do we go forward with rethinking this, what would be the action items for us?

Hi, @vustef I also agree that we should put _file in iceberg-rust, and I left some comments about how to proceed.

gbrgr · 2025-11-12T08:33:41Z

crates/iceberg/src/arrow/record_batch_transformer.rs

                let vals: Vec<Option<f64>> = vec![None; num_rows];
                Arc::new(Float64Array::from(vals))
            }
+            (DataType::RunEndEncoded(_, _), Some(PrimitiveLiteral::String(value))) => {


Should we in general encode constant columns as REE? Or should we make this custom per field? For the file path it definitely makes sense to run-end encode.

It seems a bit random to me that only strings use REE in primitive_literal_to_arrow_type below. Yes, they might use the most memory otherwise, but other types also have similar kind of redundancy suitable for the REE.

Yes that is why I am pointing it out here. If the reviewers are fine with it, I'd go with REE everywhere.

vustef

I like the new API with the with_constant. Curious to see what Renjie thinks.

vustef · 2025-11-12T08:48:51Z

crates/iceberg/src/arrow/record_batch_transformer.rs

+                // This is a virtual field - add it with the constant value
+                return Ok(ColumnSource::Add {
+                    value: Some(constant_value.clone()),
+                    target_type: Self::primitive_literal_to_arrow_type(constant_value)?,


Feels a bit redundant to have to not only lookup constants_map twice, but call primitive_literal_to_arrow_type twice (once here and once in generate_batch_transform)

I store fields now instead of constants, the double lookup is not easily avoidable in the current grand scheme of things, and should not hurt.

vustef · 2025-11-12T08:51:54Z

crates/iceberg/src/arrow/record_batch_transformer.rs

                let vals: Vec<Option<f64>> = vec![None; num_rows];
                Arc::new(Float64Array::from(vals))
            }
+            (DataType::RunEndEncoded(_, _), Some(PrimitiveLiteral::String(value))) => {


It seems a bit random to me that only strings use REE in primitive_literal_to_arrow_type below. Yes, they might use the most memory otherwise, but other types also have similar kind of redundancy suitable for the REE.

crates/iceberg/src/metadata_columns.rs

vustef · 2025-11-12T09:00:59Z

crates/iceberg/src/metadata_columns.rs

+/// The field ID of the metadata column, or an error if the column name is not recognized
+pub fn get_metadata_field_id(column_name: &str) -> Result<i32> {
+    match column_name {
+        RESERVED_COL_NAME_FILE => Ok(RESERVED_FIELD_ID_FILE),


wish that we could somehow reuse the mapping from get_metadata_column_name, to form some bidirectional 1-1 mapping. Perhaps some macro magic or something else?

Possible, but I don't think it pays off for a hand full of fields.

crates/iceberg/src/arrow/record_batch_transformer.rs

gbrgr and others added 8 commits November 4, 2025 08:38

Add REE file column helpers

aab78d6

Add helper tests

ee21cab

Add constants

37b52e2

Add support for _file constant

44463a0

Merge branch 'main' into feature/gb/file-column

b5449f6

Update tests

e034009

Fix clippy warning

4f0a4f1

Fix doc test

51f76d3

vustef reviewed Nov 4, 2025

View reviewed changes

crates/iceberg/src/arrow/reader.rs Outdated Show resolved Hide resolved

gbrgr and others added 2 commits November 4, 2025 12:52

Track in field ids

d84e16b

Merge branch 'main' into feature/gb/file-column

984dacd

gbrgr changed the title ~~Add support for _file column~~ feat(core): Add support for _file column Nov 4, 2025

gbrgr added 2 commits November 4, 2025 15:32

Add test

bd478cb

Merge branch 'feature/gb/file-column' of github.com:RelationalAI/iceb…

8593db0

…erg-rust into feature/gb/file-column

gbrgr marked this pull request as ready for review November 4, 2025 14:32

gbrgr and others added 2 commits November 4, 2025 16:04

Allow repeated virtual file column selection

9b186c7

Merge branch 'main' into feature/gb/file-column

30ae5fb

alamb mentioned this pull request Nov 5, 2025

Support extended partition cols for listing table. apache/datafusion#18482

Open

liurenjie1024 requested changes Nov 6, 2025

View reviewed changes

liurenjie1024 mentioned this pull request Nov 6, 2025

General support for metadata columns + implementation for _pos #1791

Draft

gbrgr added 2 commits November 7, 2025 14:03

Refactor into own transformer step

adf0da0

Merge branch 'feature/gb/file-column' of github.com:RelationalAI/iceb…

f4336a8

…erg-rust into feature/gb/file-column

vustef reviewed Nov 7, 2025

View reviewed changes

gbrgr added 3 commits November 7, 2025 15:04

Revert "Refactor into own transformer step"

ef3a965

This reverts commit adf0da0.

Avoid special casing in batch creation

534490b

.

04bf463

liurenjie1024 reviewed Nov 10, 2025

View reviewed changes

gbrgr added 2 commits November 12, 2025 08:57

Modify record batch transformer to support reserved fields

9e88edf

Add metadata column helper functions

060b45d

gbrgr commented Nov 12, 2025

View reviewed changes

vustef reviewed Nov 12, 2025

View reviewed changes

gbrgr added 6 commits November 12, 2025 11:16

Store fields instead of constants

8572dae

Add comment

f273add

Adapt comment

5aa92ae

.

c05b886

Adapt error message

33bb0ad

Consider field_id range

42167ff

feat(core): Add support for _file column #1824

Are you sure you want to change the base?

feat(core): Add support for _file column #1824

Uh oh!

Conversation

gbrgr commented Nov 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

What changes are included in this PR?

Key Changes

Are these changes tested?

New Tests (7 tests added)

Table Scan API Tests (7 tests)

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vustef commented Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

liurenjie1024 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liurenjie1024 commented Nov 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vustef left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

feat(core): Add support for `_file` column #1824

feat(core): Add support for `_file` column #1824

gbrgr commented Nov 4, 2025 •

edited

Loading

vustef commented Nov 6, 2025 •

edited

Loading