Skip to content

Keep rows in truncate partitions near the minimum int or long - #4407

Open
Anurag-S1ngh wants to merge 6 commits into
apache:mainfrom
Anurag-S1ngh:fix/4403-truncate-min-wrap
Open

Anurag-S1ngh wants to merge 6 commits into
apache:mainfrom
Anurag-S1ngh:fix/4403-truncate-min-wrap

Conversation

@Anurag-S1ngh

@Anurag-S1ngh Anurag-S1ngh commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The writer, like Java, truncates with two's complement arithmetic, so values from the minimum of the type wrap to a partition value at the top of the range. The Python transform did not wrap, so scan planning projected a different partition, pruned the file, and lost those rows for EqualTo, In, and range filters.

Changes

  • Wrap the Python truncate transform into the range of the source type, which fixes EqualTo and In.
  • Add the wrapped partition value to the inclusive projection of ranges that can match it. Greater-than filters use the first partition after the block that holds the minimum, because the boundary itself wraps.
  • Return no strict projection for int and long truncate ranges that can match the wrapped partition, because it does not sort with the other partition values. The residual evaluator then keeps the original predicate, so count() reads the file instead of trusting its record count.
  • Skip the inclusive projection in the residual evaluator when a transform projects to an expression instead of a single predicate. Residuals are an optimization, so falling back to the original predicate is safe.

The writer output is unchanged, matching Java and existing tables. This also fixes add_files for a Parquet file whose minimum is the type minimum: on main, adding the file raised struct.error: 'i' format requires -2147483648 <= number <= 2147483647; the file now gets the partition value the writer uses.

The wrapped-partition range projection goes beyond Java: TruncateUtil.truncateInt wraps the same way, but ProjectionUtil.truncateInteger does not add the wrapped partition, so Java has the same gap for < and >.

Testing

  • Added tests for the transform at the type minimum, the projections near the minimum, and the strict projection near the minimum.
  • Added an end-to-end scan of a truncate[1000] int partition with EqualTo, LessThan, and GreaterThan at the minimum.
  • Added count() coverage on a table with write.metadata.metrics.default=counts, where there are no column bounds.
  • make test passes (4787 passed, 5 skipped).

Closes #4403

The writer, like Java, truncates with two's complement arithmetic, so the
values from the minimum of the type wrap to a partition value at the top of
the range. The Python transform did not wrap, so scan planning projected a
different partition and pruned the file, losing those rows for EqualTo, In,
and range filters.

Wrap the Python transform into the range of the source type, and add the
wrapped partition value to the inclusive projection of ranges that can
match it. Greater-than filters need the first partition value after the
block that holds the minimum, because the boundary itself wraps. Skip the
inclusive projection in the residual evaluator when a transform projects to
an expression instead of a single predicate.
@Fokko
Fokko self-requested a review November 17, 2026 17:28

@rahulsmahadev rahulsmahadev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing this. I ran it locally against main (33b4ce5):

  • The new and changed tests fail on main and pass here.
  • I swept =, !=, <, <=, >, >=, IN and NOT IN at both ends of int and long, with truncate[1000] and truncate[7], comparing to_arrow() and count() with the expected rows: 77 wrong results on main, none here.
  • It also fixes add_files for such a file. On main, adding a Parquet file with x = -2**31 to this spec fails with struct.error: 'i' format requires -2147483648 <= number <= 2147483647; here the file gets partition 2147483296, the value the writer uses. Maybe worth a line in the description.

One problem I think should be fixed here: the strict projection still assumes that truncate partitions sort like the source values, and ResidualEvaluator trusts it, so count() is wrong for files without column bounds (for example with write.metadata.metrics.default=counts):

  • New with this change: transform(MIN + 100) is now 2147483296, so the strict projection of x < MIN + 100 is x_trunc < 2147483296. That holds for every ordinary partition, so count() adds their rows.
  • Already on main: the strict projection of x > 0 is x_trunc > 0, which holds for the wrapped partition, so count() adds the rows near the minimum.
import tempfile

import pyarrow as pa

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import GreaterThan, LessThan
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import TruncateTransform
from pyiceberg.types import IntegerType, NestedField

tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
schema = Schema(NestedField(1, "x", IntegerType()))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=TruncateTransform(1000), name="x_trunc"))
t = catalog.create_table("ns.t", schema=schema, partition_spec=spec, properties={"write.metadata.metrics.default": "counts"})
t.append(pa.table({"x": pa.array([-(2**31), 5], pa.int32())}))

for row_filter in [LessThan("x", -(2**31) + 100), GreaterThan("x", 0)]:
    scan = t.scan(row_filter=row_filter)
    print(len(scan.to_arrow()), scan.count())  # expected 1 1 for both; this PR prints 1 2 for both, main 0 0 and 1 2

to_arrow() is right in both cases because it re-applies the row filter. For < and <= with a boundary in the block of the minimum there is no safe strict projection, so strict_project could return None there; for > and >= it needs to exclude the wrapped value, or also return None. A count() assert on a table with write.metadata.metrics.default=counts would cover both.

Smaller things:

  • Java: TruncateUtil.truncateInt wraps the same way, but ProjectionUtil.truncateInteger does not add the wrapped partition, so Java has the same gap for < and >. It would help to say in the description that the range projection goes beyond Java.
  • #4401, #4402, #4406 and this PR all add tests right after test_append_to_table_bucketed_by_time and edit the same import block in tests/io/test_pyarrow.py, so each pair conflicts. Placing each test next to related tests would let them merge in any order.

Comment thread pyiceberg/transforms.py
if boundary + 1 < next_block_start:
return Or(GreaterThanOrEqual(Reference(name), next_block_start), EqualTo(Reference(name), wrapped_minimum))
return _truncate_number(name, pred, transform)
elif isinstance(pred, BoundGreaterThanOrEqual):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No test covers this branch: with if boundary < next_block_start: replaced by if False:, every test still passes, although x >= MIN then skips all ordinary partitions. An assert next to the GreaterThan one in test_int_truncate_projection_near_minimum would cover it:

assert project(GreaterThanOrEqual("id", -(2**63) + 1)) == Or(
    GreaterThanOrEqual("id_trunc", -9223372036854775800), EqualTo("id_trunc", wrapped_minimum)
)

That test projects the LongType id column, so an IntegerType case would help too; only the scan test covers int.

Comment thread pyiceberg/expressions/visitors.py Outdated
inclusive_projection = part.transform.project(part.name, predicate)
inclusive_result = None
if inclusive_projection is not None:
if isinstance(inclusive_projection, UnboundPredicate):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TruncateTransform.project() can now return an Or, but Transform.project(), TruncateTransform.project() and _truncate_integer_projection() are still annotated UnboundPredicate | None, and this guard shows that callers relied on that. Widening the annotations to BooleanExpression | None would let mypy point at callers like this one.

#4278 rewrites this block into _eval_projection(projection: UnboundPredicate | None), which calls .bind() on the projection, so whichever of the two PRs merges second needs this guard there too.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Anurag-S1ngh! Great catch, this is a subtle one: the writer (like Java) wraps the truncate of the type minimum via two's complement, so the partition value sits at the top of the range and scan planning silently pruned those files.

I pulled the branch and verified:

  • tests/test_transforms.py, tests/expressions/test_projection.py, tests/utils/test_datetime.py, tests/expressions/test_literals.py all pass
  • Reproduced the end-to-end scenario: TruncateTransform(1000) on an int column with value INT_MIN, scanned with EqualTo, In, LessThan, GreaterThan: correct rows come back
  • Checked the wrap math against Java's floorMod semantics for int32/int64 at several widths, all consistent
  • Confirmed the isinstance(inclusive_projection, UnboundPredicate) guard in visitors.py is needed and safe: the new Or(...) projections are BooleanExpressions, not predicates, and skipping the inclusive re-check there just falls back to the original predicate (which is correct, only less optimal)

One nit, non-blocking: in _truncate_integer_projection (pyiceberg/transforms.py, around line 1048), the not isinstance(pred.literal, LongLiteral) guard falls back to _truncate_number. Since bound int/long predicates always carry LongLiteral literals in practice, that branch is mostly defensive. If a DecimalLiteral ever landed here, pred.literal.value would be a Decimal and the comparison would still work, so no action needed, just noting the fallback covers it.

CI is fully green. LGTM!

Values from the minimum of the source type truncate to a partition value
at the top of the type's range, which does not sort with the other
partition values. The strict projection assumed it did, so the residual
evaluator could treat every row of a file as matching and count()
over-counted files without column bounds.

Return no strict projection for less-than predicates that can match the
wrapped partition value, and for greater-than predicates when the type
wraps at all, falling back to reading and filtering the file.
Keeping it out of the end of the partitioning group avoids conflicts with
the other branches that add tests there.
With write.metadata.metrics.default=counts there are no column bounds, so
count() relies on the residual. Assert it matches the filtered rows for
less-than and greater-than filters at the minimum.
@Anurag-S1ngh

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed check. I reproduced the count() case and fixed it in af385e0.

  • The strict projection now returns None for less-than predicates that can match the wrapped partition, and for greater-than predicates when the type wraps at all, so the residual keeps the original predicate and count() reads the file.
  • Added test_count_truncate_partitions_near_the_minimum with write.metadata.metrics.default=counts, asserting count() and to_arrow() agree for < and > at the minimum.
  • Ran a local sweep of =, !=, <, <=, >, >=, IN and NOT IN at both ends of int and long with truncate[1000] and truncate[7]: 416 checks, no mismatch between to_arrow() and count().
  • Moved the scan test next to the other partitioning tests, and updated the description with the add_files fix and the note that the range projection goes beyond Java.

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor Author

Thanks Fokko. I left the LongLiteral guard as is, per your note. The strict projection is now conservative near the wrapped partition (af385e0), since the partition value does not sort with the source values.

…in-wrap

# Conflicts:
#	pyiceberg/expressions/visitors.py
…in-wrap

# Conflicts:
#	tests/io/test_pyarrow.py
Comment thread pyiceberg/transforms.py

if isinstance(pred, BoundGreaterThan):
if boundary + 1 < next_block_start:
return Or(GreaterThanOrEqual(Reference(name), next_block_start), EqualTo(Reference(name), wrapped_minimum))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the EqualTo here is redundant. wrapped_minimum is always greater than next_block_start (they differ by the size of the type's range minus the width), so GreaterThanOrEqual(Reference(name), next_block_start) already includes it. Returning just that keeps the projection an UnboundPredicate, so the residual evaluator can still use it to prune. Same for the BoundGreaterThanOrEqual branch below.

Comment thread pyiceberg/transforms.py

projected = _truncate_number(name, pred, transform)
if isinstance(pred, (BoundLessThan, BoundLessThanOrEqual)) and projected is not None:
return Or(projected, EqualTo(Reference(name), wrapped_minimum))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: when the boundary falls in the block that holds the minimum (for example x < -2**31 + 100 with truncate[1000]), _truncate_number projects to LessThanOrEqual(name, wrapped_minimum), which keeps almost every partition. Only the wrapped partition can match there, so this could return EqualTo(Reference(name), wrapped_minimum) alone. That is the case test_count_truncate_partitions_near_the_minimum exercises.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

truncate partitions lose rows near the minimum int or long

3 participants