Repository navigation
Add compute_partition_statistics() to MaintenanceTable - #4423
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
Add table.maintenance.compute_partition_statistics(), which computes the partition statistics of a snapshot from its manifests, writes them to a Parquet file and registers it, so users need not build the file by hand (follow-up to apache#4397). Full compute only, as in Java's PartitionStatsHandler. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@ebyhr, this builds on the partition-statistics computation you proposed in #3471 and follows up on Fokko's question in #4397 about having PyIceberg compute the partition statistics. @kevinjqliu, you reviewed the maintenance API in #2143. Could you take a look when you have a chance? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Follow-up to #4397. PyIceberg can register a partition statistics file but can't compute one, so users have to write the file themselves. This PR adds
table.maintenance.compute_partition_statistics(). It computes the stats from the snapshot's manifests, writes the Parquet file undermetadata/and registers it, like Java'sPartitionStatsHandlerplus Spark'scompute_partition_statsprocedure.ebyhr proposed something similar in #3471 (closed).
Same as Java:
last_updated_atandlast_updated_snapshot_id.Not supported yet:
NotImplementedError) and tables whosewrite.format.defaultis not Parquet (ValueError).ArrowNotImplementedError, same asinspect.partitions(), see Fix inspect file and partition tables for UUID columns #4358).I put this under
table.maintenancenext toexpire_snapshots(), since that's where Java's docs put it. Happy to move it toTable, or make it compute only, if you prefer.This also fixes the
Table.update_partition_statistics()docstring example from #4397.Are these changes tested?
Yes, added tests in
tests/table/test_statistics.pythat run on the memory, sql and sql_without_rowcount catalogs. They cover appends, overwrites and deletes, multiple partition specs, partition value types including float NaN and -0.0, delete files, expired snapshots, the file layout and the error cases. They also pass on PyArrow 18.0.0.Are there any user-facing changes?
Yes, new API
MaintenanceTable.compute_partition_statistics(snapshot_id=None), documented under "Table Maintenance" in the API docs.Please add the
changeloglabel.This pull request and its description were written by Isaac.