Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance issue with range queries over a partitioned table. #811

Open
ryzhyk opened this issue Dec 16, 2024 · 2 comments · May be fixed by #821
Open

Performance issue with range queries over a partitioned table. #811

ryzhyk opened this issue Dec 16, 2024 · 2 comments · May be fixed by #821

Comments

@ryzhyk
Copy link
Contributor

ryzhyk commented Dec 16, 2024

I ran into a performance issue querying an Iceberg table in S3 via the datafusion provider. The table was created using pyiceberg with the following schema:

schema = Schema(
    NestedField(1, "id", LongType(), required=True),
    NestedField(2, "name", StringType(), required=False),
    NestedField(3, "b", BooleanType(), required=True),
    NestedField(4, "ts", TimestampType(), required=True),
    NestedField(5, "dt", DateType(), required=True),
)

The table is partitioned by date extracted from the ts column:

partition_spec = PartitionSpec(
    PartitionField(
        source_id=4, field_id=1000, transform=DayTransform(), name="date"
    )
)

There are 10,000,000 records in the table spread evenly across ~200 partitions for dates between 2023-01-01 and 2023-08-02.

I query the table using iceberg-rust via the datafusion table provider using range queries of the form:

select * from my_table where ts >= timestamp '2023-01-05T00:00:00' and ts < timestamp '2023-01-06T00:00:00'

I expect this query to be very efficient, as it only needs to read one partition, however in reality it takes about as long as scanning the entire table with select * from my_table (approximately 10 seconds). It looks like predicate pushdown doesn't work here for some reason.

Questions:

  • Is this a performance issue in iceberg-rust or am I doing something wrong?
  • Is there a better way to perform this query efficiently?

I am using the latest main branch of this repo.

Thanks in advance!
ryzhyk pushed a commit to feldera/feldera that referenced this issue Dec 17, 2024
[adapters] Iceberg source connector.
55e1585 
Initial implementation of the Iceberg source connector. The connector is
built on the `iceberg` crate, which still in its early days and has many
limitations and performance issues.

* It currently only supports primitive types (no structs, maps, lists)
* It only supports reading tables (hence no sink connector yet)
* It only supports snapshot reads, not table following, although I think
  the latter could be mostly implemented using available low-level APIs.
* I haven't figured out how to do efficient range queries for time
  seried data: apache/iceberg-rust#811

The implementation has a very similar structure to the Delta Lake
connector and actually share a bunch of code with it (I moved some of
this code to `adapterslib`, but I copied some other code, which I
thought may diverge in the future).  Both connectors register the table
as a datafusion table provider and mostly work with it via the
datafusion API.

The main difference between Iceberg and Delta is that Iceberg cannot really
be used without a catalog, since catalog is responsible for tracking the
location of the latest metadata file (metadata file is the root object
required to do anything with the Iceberg table). We currently support
two of the most common catalog APIs: Glue (for Iceberg tables in AWS),
and REST, which seems to be increasingly popular in the Iceberg
community. We should be able to easily add SQL and hive catalogs, which
are supported by the `iceberg` crate.

The connector should work with tables in S3, local FS, and GCS, but only the
first two have been tested. The `iceberg` crate currently doesn't
support azure and other data stores, although it should be easy to add them if
necessary, since they are supported by the `opendal` crate, which
`iceberg` uses for FileIO.

Signed-off-by: Leonid Ryzhyk <[email protected]>
ryzhyk pushed a commit to feldera/feldera that referenced this issue Dec 17, 2024
[adapters] Iceberg source connector.
d642707 
Initial implementation of the Iceberg source connector. The connector is
built on the `iceberg` crate, which still in its early days and has many
limitations and performance issues.

* It currently only supports primitive types (no structs, maps, lists)
* It only supports reading tables (hence no sink connector yet)
* It only supports snapshot reads, not table following, although I think
  the latter could be mostly implemented using available low-level APIs.
* I haven't figured out how to do efficient range queries for time
  seried data: apache/iceberg-rust#811

The implementation has a very similar structure to the Delta Lake
connector and actually share a bunch of code with it (I moved some of
this code to `adapterslib`, but I copied some other code, which I
thought may diverge in the future).  Both connectors register the table
as a datafusion table provider and mostly work with it via the
datafusion API.

The main difference between Iceberg and Delta is that Iceberg cannot really
be used without a catalog, since catalog is responsible for tracking the
location of the latest metadata file (metadata file is the root object
required to do anything with the Iceberg table). We currently support
two of the most common catalog APIs: Glue (for Iceberg tables in AWS),
and REST, which seems to be increasingly popular in the Iceberg
community. We should be able to easily add SQL and hive catalogs, which
are supported by the `iceberg` crate.

The connector should work with tables in S3, local FS, and GCS, but only the
first two have been tested. The `iceberg` crate currently doesn't
support azure and other data stores, although it should be easy to add them if
necessary, since they are supported by the `opendal` crate, which
`iceberg` uses for FileIO.

Signed-off-by: Leonid Ryzhyk <[email protected]>
@ryzhyk ryzhyk mentioned this issue Dec 17, 2024
Merged
ryzhyk pushed a commit to feldera/feldera that referenced this issue Dec 17, 2024
adapters: Iceberg source connector.
9ad8546 
Initial implementation of the Iceberg source connector. The connector is
built on the `iceberg` crate, which still in its early days and has many
limitations and performance issues.

* It currently only supports primitive types (no structs, maps, lists)
* It only supports reading tables (hence no sink connector yet)
* It only supports snapshot reads, not table following, although I think
  the latter could be mostly implemented using available low-level APIs.
* I haven't figured out how to do efficient range queries for time
  seried data: apache/iceberg-rust#811

The implementation has a very similar structure to the Delta Lake
connector and actually share a bunch of code with it (I moved some of
this code to `adapterslib`, but I copied some other code, which I
thought may diverge in the future).  Both connectors register the table
as a datafusion table provider and mostly work with it via the
datafusion API.

The main difference between Iceberg and Delta is that Iceberg cannot really
be used without a catalog, since catalog is responsible for tracking the
location of the latest metadata file (metadata file is the root object
required to do anything with the Iceberg table). We currently support
two of the most common catalog APIs: Glue (for Iceberg tables in AWS),
and REST, which seems to be increasingly popular in the Iceberg
community. We should be able to easily add SQL and hive catalogs, which
are supported by the `iceberg` crate.

The connector should work with tables in S3, local FS, and GCS, but only the
first two have been tested. The `iceberg` crate currently doesn't
support azure and other data stores, although it should be easy to add them if
necessary, since they are supported by the `opendal` crate, which
`iceberg` uses for FileIO.

Signed-off-by: Leonid Ryzhyk <[email protected]>
ryzhyk pushed a commit to feldera/feldera that referenced this issue Dec 18, 2024
adapters: Iceberg source connector.
f7460d0 
Initial implementation of the Iceberg source connector. The connector is
built on the `iceberg` crate, which still in its early days and has many
limitations and performance issues.

* It currently only supports primitive types (no structs, maps, lists)
* It only supports reading tables (hence no sink connector yet)
* It only supports snapshot reads, not table following, although I think
  the latter could be mostly implemented using available low-level APIs.
* I haven't figured out how to do efficient range queries for time
  seried data: apache/iceberg-rust#811

The implementation has a very similar structure to the Delta Lake
connector and actually share a bunch of code with it (I moved some of
this code to `adapterslib`, but I copied some other code, which I
thought may diverge in the future).  Both connectors register the table
as a datafusion table provider and mostly work with it via the
datafusion API.

The main difference between Iceberg and Delta is that Iceberg cannot really
be used without a catalog, since catalog is responsible for tracking the
location of the latest metadata file (metadata file is the root object
required to do anything with the Iceberg table). We currently support
two of the most common catalog APIs: Glue (for Iceberg tables in AWS),
and REST, which seems to be increasingly popular in the Iceberg
community. We should be able to easily add SQL and hive catalogs, which
are supported by the `iceberg` crate.

The connector should work with tables in S3, local FS, and GCS, but only the
first two have been tested. The `iceberg` crate currently doesn't
support azure and other data stores, although it should be easy to add them if
necessary, since they are supported by the `opendal` crate, which
`iceberg` uses for FileIO.

Signed-off-by: Leonid Ryzhyk <[email protected]>
Fokko added a commit to Fokko/iceberg-rust that referenced this issue Dec 18, 2024
@Fokko
Datafusion: Support cast operations
13798a3 
Resolves apache#811

Depends on apache#820
Fokko added a commit to Fokko/iceberg-rust that referenced this issue Dec 18, 2024
@Fokko
Datafusion: Support cast operations
2b20f79 
Resolves apache#811

Depends on apache#820
Fokko added a commit to Fokko/iceberg-rust that referenced this issue Dec 18, 2024
@Fokko
Datafusion: Support cast operations
24d103d 
Resolves apache#811

Depends on apache#820
@Fokko Fokko linked a pull request Dec 18, 2024 that will close this issue
Draft
@Fokko
Copy link
Contributor

Fokko commented Dec 18, 2024 •
edited
Loading

Thanks for reporting this @ryzhyk. I did take a look and it looks like what I already suspected, it does not take the cast from string to datetime into consideration. This means that it will fallback to a full table scan.

Fokko added a commit to Fokko/iceberg-rust that referenced this issue Dec 18, 2024
@Fokko
Datafusion: Support cast operations
073e4ee 
Resolves apache#811

Depends on apache#820
@ryzhyk
Copy link
Contributor Author

ryzhyk commented Dec 18, 2024

Many thanks for investigating the issue and for making a fix. That was really fast!

ryzhyk pushed a commit to feldera/feldera that referenced this issue Dec 18, 2024
adapters: Iceberg source connector.
1fe8ebe 
Initial implementation of the Iceberg source connector. The connector is
built on the `iceberg` crate, which still in its early days and has many
limitations and performance issues.

* It currently only supports primitive types (no structs, maps, lists)
* It only supports reading tables (hence no sink connector yet)
* It only supports snapshot reads, not table following, although I think
  the latter could be mostly implemented using available low-level APIs.
* I haven't figured out how to do efficient range queries for time
  seried data: apache/iceberg-rust#811

The implementation has a very similar structure to the Delta Lake
connector and actually share a bunch of code with it (I moved some of
this code to `adapterslib`, but I copied some other code, which I
thought may diverge in the future).  Both connectors register the table
as a datafusion table provider and mostly work with it via the
datafusion API.

The main difference between Iceberg and Delta is that Iceberg cannot really
be used without a catalog, since catalog is responsible for tracking the
location of the latest metadata file (metadata file is the root object
required to do anything with the Iceberg table). We currently support
two of the most common catalog APIs: Glue (for Iceberg tables in AWS),
and REST, which seems to be increasingly popular in the Iceberg
community. We should be able to easily add SQL and hive catalogs, which
are supported by the `iceberg` crate.

The connector should work with tables in S3, local FS, and GCS, but only the
first two have been tested. The `iceberg` crate currently doesn't
support azure and other data stores, although it should be easy to add them if
necessary, since they are supported by the `opendal` crate, which
`iceberg` uses for FileIO.

Signed-off-by: Leonid Ryzhyk <[email protected]>
github-merge-queue bot pushed a commit to feldera/feldera that referenced this issue Dec 18, 2024
@ryzhyk
adapters: Iceberg source connector.
f2bfb87 
Initial implementation of the Iceberg source connector. The connector is
built on the `iceberg` crate, which still in its early days and has many
limitations and performance issues.

* It currently only supports primitive types (no structs, maps, lists)
* It only supports reading tables (hence no sink connector yet)
* It only supports snapshot reads, not table following, although I think
  the latter could be mostly implemented using available low-level APIs.
* I haven't figured out how to do efficient range queries for time
  seried data: apache/iceberg-rust#811

The implementation has a very similar structure to the Delta Lake
connector and actually share a bunch of code with it (I moved some of
this code to `adapterslib`, but I copied some other code, which I
thought may diverge in the future).  Both connectors register the table
as a datafusion table provider and mostly work with it via the
datafusion API.

The main difference between Iceberg and Delta is that Iceberg cannot really
be used without a catalog, since catalog is responsible for tracking the
location of the latest metadata file (metadata file is the root object
required to do anything with the Iceberg table). We currently support
two of the most common catalog APIs: Glue (for Iceberg tables in AWS),
and REST, which seems to be increasingly popular in the Iceberg
community. We should be able to easily add SQL and hive catalogs, which
are supported by the `iceberg` crate.

The connector should work with tables in S3, local FS, and GCS, but only the
first two have been tested. The `iceberg` crate currently doesn't
support azure and other data stores, although it should be easy to add them if
necessary, since they are supported by the `opendal` crate, which
`iceberg` uses for FileIO.

Signed-off-by: Leonid Ryzhyk <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(datafusion): Support cast operations Fokko/iceberg-rust
2 participants
@Fokko @ryzhyk