Minimum Viable Product (MVP) Roadmap

[wgtmac]

Below are planned roadmap for MVP as discussed in different places (e.g. dev ML, Github issues & PRs, slack channel, etc.). Note that it is only for the native C++ implementation. For the effort of Rust C++ binding, please refer to https://lists.apache.org/thread/hotlcdw86nrmt7cf5o5o7kq6gwo98758.

Convention

• Platform: Linux, MacOS, Windows
• Compilers: Clang, GCC, MSVC
• Build: CMake
• C++ standard: C++20
• Coding style: Follow what Apache Arrow C++ does: https://arrow.apache.org/docs/developers/cpp/development.html#code-style-linting-and-ci
• Error handling: use ported expected similar to std::expected

Goal

• Implement read path for parsing metadata files of Iceberg v1 & v2. It is a nice-to-have feature to read data files depending on the bandwidth of contributors.
• Provide a light-weight io-less iceberg library with minimal dependencies (like apache/nanoarrow and nlohmann/json) to mainly deal with the Iceberg metadata. Downstream projects are required to provide their own implementations like I/O, Parquet, Avro and write adaptation code.
• Provide a battery-included iceberg-bundle library backed by Apache Arrow C++ and Apache Avro C++ libraries.

Workitems

(Disclaimer: this is not an exhaustive list and is subject to change as the development goes on)

API of metadata or building block

• Add Schema (including data types)
• Add DataFile
• Add DeleteFile
• Add ManifestFile
• Add ManifestEntry @zhjwpku
• Add Snapshot @zhjwpku
• Add PartitionSpec
• Add SortOrder @zhjwpku
• Add ManifestList
• Add TableMetadata

Catalog

• Define Catalog interface.
• Implement an in-memory catalog. @gty404

IO

• Define FileIO interface with minimal operations.
• Provide default FileIO implementation backed by arrow::FileSystem for different storage providers.

Table

• Define Table interface.
• Provide a basic Table implementation to have access to its metadata. @lishuxu
• Implement Table::NewScan function and TableScan class to support planning files for reading a specific snapshot. @gty404
• Implement partition pruning and data file pruning in the TableScan.

JSON Serialization

• SortOrder @gty404
• PartitionSpec @gty404
• Schema
• Snapshot
• TableMetadata
• NameMapping

Metadata File Reader

• JSON file reader.
• JSON parser for metadata objects.
• Reading gzip-compressed metadata json file.

File Format Reader

• Define format-agnostic FileReader interface with Arrow C Data as the contract.
• Implement manifest list file reader.
• Implement manifest file reader.
• Provide default Avro reader implementation in the iceberg-bundle library.
• Provide default Parquet reader implementation in the iceberg-bundle library.

Schema/Data conversion

• Bi-directional conversion to Arrow C schema.
• ArrowArray -> avro::GenericDatum
• avro::GenericDatum -> ArrowArray
• StructLike interface

Expression

• Transform metadata
• Transform function
• Expression components: expression, term, literal, reference, etc.
• Expression visitor

Third-party library

• Add nanoarrow to libiceberg
• Add nlohmann/json to libiceberg @yingcai-cy
• Add avro-cpp to libiceberg-bundle
• Add arrow-cpp to libiceberg-bundle

First release

• Check licenses of dependencies.
• Add and check documentations.
• Add release script.

[wgtmac]

I have made a bold suggestion that the type system to directly leverage Arrow C++ to avoid re-invent the wheels and benefit from RecordBatch, Expression and other stuff. I saw that iceberg-rust and iceberg-go have implemented its own data types. Is there any issue that arrow type system is unable to deal with iceberg type system? @Xuanwo @zeroshade

[zeroshade]

The biggest drawback to just using the Arrow C++ type system directly is that the mappings aren't perfect for iceberg.

Iceberg only has Int32 and Int64 while Arrow has Int 8/16/32/64 and Uint 8/16/32/64. The same goes for all of the other types that exist in Arrow but don't exist for Iceberg (such as the Large* variants, REE, and so on). Another issue is how Time and Timestamp types are handled: Iceberg defines the unit to be milliseconds while Arrow parameterizes the types. For the most part you can see the logic needed for converting between Iceberg and Arrow type systems here

The differences in the types means that even if you re-use the types from Arrow, you're still going to eventually have to perform a conversion / implement this logic when it comes to reading/writing data and converting it to Arrow. This is why I provided functions to convert an Arrow Schema to Iceberg and vice-versa in the iceberg-go library. Reading data still returns a stream of Arrow record batches, and when I implement writing, it'll accept a stream of Arrow record batches to write.

It's not that there's specific issues the Arrow type system can't deal with, it's more that there are significantly more types and flexibility in the Arrow type system than what is available in the Iceberg type system.

[wgtmac]

Thanks @zeroshade for the detail!

The table below is the type mapping between iceberg and arrow. I think we can provide a wrapper around arrow data types to use only a subset of them. On the read path, the mapping is pretty clear except for String/LargeString/Binary/LargeBinary. We can by default use String/Binary unless explicitly configured. On the write path, we can simply error out for unsupported arrow types. Just want to add that the ongoing iceberg variant and geometry types will not have any issue, parquet-cpp will anyway implement them because they are part of the parquet spec. Therefore I don't think there is a compelling reason not to use arrow::DataType directly.

iceberg	arrow
unknown	Null
boolean	Boolean
int	Int32
long	Int64
float	Float32
double	Float64
decimal(P,S)	Decimal(P,S)
date	Date32
time	Time64
timestamp	Timestamp(MICRO)
timestamptz	Timestamp(MICRO,UTC)
timestamp_ns	Timestamp(NANO)
timestamptz_ns	Timestamp(NANO,UTC)
string	String/LargeString
uuid	UUID canonical extension type
fixed(L)	FixedSizeBinary (L)
binary	Binary/LargeBinary
struct	Struct
list	List/LargeList
map	Map

[zeroshade]

For Arrow decimal types you'll need to specify which decimal type to use, I recommend using 128-bit because that's the max supported by iceberg.

For the geometry type, you can use the GeoArrow community extension types.

There is also a proposal to add the variant type to Arrow also after it is accepted into parquet, so that should work out fine too.

[wgtmac]

Agreed, in essence this is pretty much the same question to parquet-cpp on what is the best arrow data type to use for a specific parquet (iceberg) type.

[Fokko]

Hey everyone, and thanks @wgtmac for kickstarting the discussion. Sharing my thoughts below:

Types

I would also lean towards having a separate type system. Like @zeroshade already pointed out, for writing a decimal into Parquet, there are certain mapping that need to be followed according to the spec. Another issue that I ran into with PyIceberg, is the limited support for Parquet Field-IDs in Arrow, this is something that Iceberg heavily relies on. For Arrow this is stored as a binary field in the metadata, with Iceberg we often traverse over the schema would incur a lot of (de)serialization. Also, for the field, things like the initial- and write default need to be tracked, which is not part of Arrow currently. Therefore having schema primitives specifically for Iceberg makes it easier, and like Matt mentioned, it is easy to convert the one to the other.

Format

data representation: row-wise (Avro record?) and columnar (arrow::RecordBatch?)

I think we need both. Metadata is encoded in Avro, and for the data itself, the majority is in Parquet. Iceberg also supports Avro and ORC for storing data, but that's only being used by a fraction of the community.

IO

For IO there is an opinionated approach within Iceberg, called the FileIO: https://github.com/apache/iceberg/blob/f7ff0dc8c0a27e2bcd727e4f7705cf0a69ccc9b3/api/src/main/java/org/apache/iceberg/io/FileIO.java#L29-L36

This implements all the reading that's being used for Iceberg. One important distinction with a traditional filesystem is that it doesn't support listing and moving of files, therefore being very efficient to operate on an object store. I think we can wrap the arrow::FileSystem within a FileIO (similar to what we do in PyIceberg), but I would strongly recommend also adopting this concept within Iceberg-CPP because it makes the integration much easier, for example, it also standardizes the configuration across implementations, and people could even provide their own FileIO through configuration if they like.

[wgtmac]

Thanks @Fokko for the reply!

I would also lean towards having a separate type system.

I agree that the efficiency of field-id and inability to set default value make arrow schema not that appealing. I think the problem is arrow::Field not arrow::DataType. What about creating yet another iceberg::Field to wrap arrow::DataType with better support of iceberg concepts? In this way, the type visitors of arrow-cpp still work and so do arrow::Expression and arrow::Scalar. @Fokko @zeroshade

I think we can wrap the arrow::FileSystem within a FileIO

Yes, it is exactly what's in my mind.

people could even provide their own FileIO through configuration if they like.

This might be challenging since we don't have an easy approach to use reflection in C++.

[boroknagyz]

people could even provide their own FileIO through configuration if they like.

This might be challenging since we don't have an easy approach to use reflection in C++.

I think we just need a plugin mechanism to allow that, no need for reflection. If we have a clear interface of a FileIO, it shouldn't be too hard to make things pluggable.
It can be important for S3, as some implementations (e.g. Hadoop's S3AFileSystem) use s3a:// prefixes, while others use s3:// prefixes.

+1 on having a separate type system for better control and simplicity.

[Fokko]

What about creating yet another iceberg::Field to wrap arrow::DataType with better support of iceberg concepts?

I think it should then wrap a arrow: Field, otherwise we would not be able to pass it into a arrow::Struct. But now the struct returns arrow:Field, so then we should create a iceberg::Struct as well, otherwise we're casting all over the place. My biggest concern is that we have code all over the place to coerce the Arrow superset into the Iceberg types. To me, the Arrow system mixes both how it is encoded (large_) and the actual types, while a Decimal in Iceberg should be encoded as a int32 when P <= 9.

If we have a clear interface of a FileIO, it shouldn't be too hard to make things pluggable.

That's exactly what I meant, thanks for clarifying :)

[Xuanwo]

It is interesting that iceberg-cpp can provide a header while its implementation is handled by iceberg-rust. For instance, there is an ongoing PR for puffin support: apache/iceberg-rust#714

iceberg-rust has built something similar for iceberg-python. I believe this would also be valuable for iceberg-cpp.

[pitrou]

• Dependencies: Arrow, Avro, ORC, simdjson, etc.

Dependency management in Arrow C++ has been a huge headache (and still is). I'd heavily recommend that Iceberg C++ starts with a minimal set of dependencies. I don't know what ORC or Avro has to do with it, for example.

Arrow C++ is certainly a requirement; it will give access to useful base features (IO etc.), and of course to Parquet C++ APIs.

simdjson seems reasonable as well, and a JSON library is very useful for a bunch of tasks (such as writing nice testing helpers or CLI utilities).

[wgtmac]

@pitrou

I'd heavily recommend that Iceberg C++ starts with a minimal set of dependencies.

Cannot agree more.

I don't know what ORC or Avro has to do with it, for example.

ORC can be postponed to add considering of its popularity. Avro is a must because Iceberg spec uses Avro to store its manifest file which is the home to file list and file metadata.

[lidavidm]

I would almost rather not depend on Arrow C++ if possible (what if I want to use the cuDF parquet reader, or OpenDAL for S3 access?)

[wgtmac]

@lidavidm Great question!

I would almost rather not depend on Arrow C++ if possible

AFAIK, Arrow C++ includes the most complete implementation of Parquet reader and writer. In addition, arrow columnar format seems to be the best choice to be integrated with other engines.

what if I want to use the cuDF parquet reader, or OpenDAL for S3 access?

We should design a good interface for file reader and writer so we get the chance to plugin different parquet implementations. Similarly a good FileIO abstraction does not block us to choose S3 access from Arrow FileSystem or OpenDAL implementation. Perhaps @etseidl and @Xuanwo could chime in for cuDF and OpenDAL respectively.