API, Core, Spark: Scan API for partition stats #14508
Conversation
|
Some background: The current the way to query partition stats is through The content of this PR:
These could possibly be some follow-up steps:
|
| value, | ||
| (existingEntry, newEntry) -> { | ||
| existingEntry.appendStats(newEntry); | ||
| ((PartitionStats) existingEntry).appendStats(newEntry); |
There was a problem hiding this comment.
If PartitionStatistics interface had the appendStats function, this cast (and another occurrence) wouldn't be needed. It seemed a bit weird to have it there, but I'm open to make this change to clean up casts.
There was a problem hiding this comment.
I would prefer to keep the interface clean
| PartitionStatisticsScan filter(Expression filter); | ||
|
|
||
| /** | ||
| * Create a new scan from this with the schema as its projection. |
There was a problem hiding this comment.
Maybe describe what will happen with the PartitionStatistics attributes which are not part of the schema.
| /** | ||
| * Create a new scan from this with the schema as its projection. | ||
| * | ||
| * @param schema a projection schema |
There was a problem hiding this comment.
How does the user create the Schema?
I would prefer something like the DataFile where the possible columns are available as constants, and the type is available as well. Maybe copy/move/deprecate the schema from the old place.
| Types.StructType partitionType = Partitioning.partitionType(table); | ||
| Schema schema = PartitionStatsHandler.schema(partitionType, TableUtil.formatVersion(table)); | ||
|
|
||
| FileFormat fileFormat = FileFormat.fromFileName(statsFile.get().path()); |
There was a problem hiding this comment.
I still think that getting the file format from the name is brittle. Maybe not in this PR, but I would love to have this fixed
|
|
||
| try { | ||
| stats = computeAndMergeStatsIncremental(table, snapshot, partitionType, statisticsFile); | ||
| stats = computeAndMergeStatsIncremental(table, snapshot, statisticsFile.snapshotId()); |
There was a problem hiding this comment.
Why do we remove the partitionType parameter? We recalculate it later again for every file. Isn't that unnecessary?
| rec -> { | ||
| return (PartitionStats) recordToPartitionStats(rec); | ||
| }); |
| } | ||
|
|
||
| @Override | ||
| public CloseableIterable<PartitionStatistics> scan() { |
2 participants
No description provided.