Skip to content

Add compute_partition_statistics() to MaintenanceTable - #4423

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/compute-partition-statistics
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/compute-partition-statistics

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Follow-up to #4397. PyIceberg can register a partition statistics file but can't compute one, so users have to write the file themselves. This PR adds table.maintenance.compute_partition_statistics(). It computes the stats from the snapshot's manifests, writes the Parquet file under metadata/ and registers it, like Java's PartitionStatsHandler plus Spark's compute_partition_stats procedure.

stats_file = table.maintenance.compute_partition_statistics()
table.metadata.partition_statistics  # [stats_file]

ebyhr proposed something similar in #3471 (closed).

Same as Java:

  • Deleted entries only update last_updated_at and last_updated_snapshot_id.
  • NaN values form one partition, -0.0 and 0.0 stay separate, and rows are sorted with NULL first and NaN last.
  • A removed v1 partition field keeps its transform's output type.
  • If the snapshot already has a stats file, that file is returned.

Not supported yet:

  • Incremental compute. Java reuses an earlier stats file when it can.
  • v3 tables (NotImplementedError) and tables whose write.format.default is not Parquet (ValueError).
  • UUID partition fields (ArrowNotImplementedError, same as inspect.partitions(), see Fix inspect file and partition tables for UUID columns #4358).
  • Partitions that only differ in a dropped source column end up in one row. Java keeps them apart.

I put this under table.maintenance next to expire_snapshots(), since that's where Java's docs put it. Happy to move it to Table, or make it compute only, if you prefer.

This also fixes the Table.update_partition_statistics() docstring example from #4397.

Are these changes tested?

Yes, added tests in tests/table/test_statistics.py that run on the memory, sql and sql_without_rowcount catalogs. They cover appends, overwrites and deletes, multiple partition specs, partition value types including float NaN and -0.0, delete files, expired snapshots, the file layout and the error cases. They also pass on PyArrow 18.0.0.

Are there any user-facing changes?

Yes, new API MaintenanceTable.compute_partition_statistics(snapshot_id=None), documented under "Table Maintenance" in the API docs.

Please add the changelog label.

This pull request and its description were written by Isaac.

Add table.maintenance.compute_partition_statistics(), which computes the
partition statistics of a snapshot from its manifests, writes them to a
Parquet file and registers it, so users need not build the file by hand
(follow-up to apache#4397). Full compute only, as in Java's PartitionStatsHandler.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@ebyhr, this builds on the partition-statistics computation you proposed in #3471 and follows up on Fokko's question in #4397 about having PyIceberg compute the partition statistics. @kevinjqliu, you reviewed the maintenance API in #2143. Could you take a look when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant