Repository navigation
Fix scan filters at the edges of the date and float ranges - #4406
Anurag-S1ngh wants to merge 5 commits into
Conversation
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks! I ran this locally against main (33b4ce5):
- The new tests fail on main and pass here (the parametrized cases inside the date range pass on both, as expected).
_civil_from_daysmatchesdatetime.datefor every day from 0001-01-01 to 9999-12-31, so results in that range are unchanged.- The float change also fixes deletes and range filters with an infinity. On main,
x < infbinds toAlwaysTrue, sotable.delete(LessThan("x", math.inf))on a float column deletes every row, including the inf, NaN and null rows; here it keeps exactly those three.<,<=,>and>=with an infinity all change this way, so they are worth listing in the description, with a test.
The one bug I found is in the negative-era branch of _civil_from_days (inline comment). A few more points:
- Java's
DoubleLiteral.to(FLOAT)inLiterals.javahas the same range check, so Java bindsfloat_col = <double infinity>toalwaysFalseas well. Worth saying in the description that this differs from Java. StringLiteral.to(FloatType)applies the same check, soEqualTo("x", "Infinity")still returns no rows andIn("x", ["Infinity", "1.5"])only1.5. Fine as a follow-up, but it is the same bug.- Nit:
2932897 * day_nanosintest_years_and_months_beyond_the_date_rangeis beyond int64, so notimestamp_nscan hold it (they end in 2262); that assert pins an unreachable case. - This conflicts with #4407 (and #4401, #4402) in
tests/io/test_pyarrow.py, since they all add tests right aftertest_append_to_table_bucketed_by_timeand edit the same import block.
| https://howardhinnant.github.io/date_algorithms.html#civil_from_days. | ||
| """ | ||
| z = days + 719468 | ||
| era = (z if z >= 0 else z - 146096) // 146097 |
There was a problem hiding this comment.
This is the C++ trick for floor division, but Python's // already floors, so for z < 0 (days before 0000-03-01) era comes out one too small. About 3% of those days get the wrong month, and some the wrong year. For example, -0001-12-31 (days=-719529) gives year -1970 and month -23640 instead of -1971 and -23641, while the writer puts that date in year partition -1971. So EqualTo("d", -719529) on a year-partitioned table projects to d_year == -1970 and prunes the file that holds the row; main raises OverflowError there instead.
| era = (z if z >= 0 else z - 146096) // 146097 | |
| era = z // 146097 |
With this change the function matches datetime on the whole representable range and gives the same values as Java's DateTimeUtil.daysToYears / daysToMonths before year 1, and the tests in this PR still pass. None of the new test days has z < 0; adding (-719529, -1971, -23641) to test_days_to_years_and_months would pin it.
There was a problem hiding this comment.
Fixed in 16903cf: era = z // 146097. Added (-719529, -1971, -23641) to test_days_to_years_and_months, so the z < 0 branch is pinned now.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh! Two real user-facing bugs fixed here, and the 9999-12-31 "no end date" case is exactly the kind of value that shows up in production data.
I pulled the branch and verified:
tests/test_transforms.py,tests/expressions/test_projection.py,tests/utils/test_datetime.py,tests/expressions/test_literals.pyall pass- Brute-forced
_civil_from_daysagainst Python'sdatetimeover 20k random days in the valid range: identical year/month everywhere - Compared old vs new
days_to_months/micros_to_months/nanos_to_*over 5k random samples: results identical inside the representable range, so no behavior change for existing data - Reproduced both scan scenarios end to end:
EqualTo("x", inf)on a float column now returns the right row, and a year-partitioned table with9999-12-31scans correctly withLessThanOrEqualandGreaterThan - Checked pre-epoch timestamps (e.g. 1969-12-31 23:59:59.999999): months/years come out the same as before
One small question, non-blocking: in pyiceberg/utils/datetime.py, micros_to_months(micros) now does micros // 86_400_000_000 first. Since that's floor division, negative micros (pre-1970) floor toward negative infinity, which matches the old day-boundary behavior. Just double-checked it, all good.
CI is fully green. LGTM!
|
Thanks for the review. Addressed in 16903cf, 60759e1 and eb30c12:
|
|
Thanks for checking. Nothing to change there, floor division for negative micros matches the previous day-boundary behavior. |
Year and month partitions raised OverflowError when a filter projected past 9999-12-31 or before 0001-01-01, because the transform converted the day count to a datetime.date. Compute years and months from the day count with integer arithmetic, so projections just past the last representable date produce year 10000 and match no partition instead of raising. A Python float became a DoubleLiteral, and converting that to a float column turned inf into FloatAboveMax or FloatBelowMin, which bound the predicate to AlwaysFalse or AlwaysTrue. inf is a valid float value, so convert it to a FloatLiteral like any other float that fits.
The C++ algorithm truncates toward zero and compensates with z - 146096, but Python's // already floors. The correction made the era one too small for days before 0000-03-01, so year and month partitions before year 1 projected to the wrong partition.
A timestamp_ns cannot hold 2932897 days, so the asserts covered an input no table can contain.
A double infinity on a float column now binds as a float literal, so <, <=, > and >= match rows instead of binding to always true or always false.
68dc05b to
eb30c12
Compare
Summary
Filtered scans raised or returned wrong rows at two kinds of edge values.
OverflowErrorwhen a filter projected past 9999-12-31 or before 0001-01-01, because the transform converted the day count to adatetime.date. 9999-12-31 is a common "no end date" value.EqualTo,In,NotEqualTo, and the range comparisons (<,<=,>,>=) oninfreturned wrong rows for float columns. A Python float becomes aDoubleLiteral, and converting that to a float column turnedinfintoFloatAboveMaxorFloatBelowMin, which bound the predicate toAlwaysFalseorAlwaysTrue.Changes
infand-infto aFloatLiteralfor float columns, like any other float that fits. For float columns this also changes how the range comparisons bind:<and<=withinfno longer match every row, and>and>=with-infno longer match every row. For example,table.delete(LessThan("x", math.inf))used to delete every row, including the inf, NaN and null rows, and now leaves exactly those rows.Notes
DoubleLiteral.to(FLOAT), so Java bindsfloat_col = <double infinity>toalwaysFalse. This change intentionally diverges from that.StringLiteral.to(FloatType)still turns values such as"Infinity"intoFloatAboveMax, soEqualTo("x", "Infinity")returns no rows. That is the same bug through the string path and is left for a follow-up.Testing
inf,Inand the range comparisons on a float column, and a year-partitioned table scanned at 9999-12-31.tests/utils/test_datetime.py,tests/expressions/test_literals.py,tests/expressions/test_projection.pyand the non-cloud tests intests/io/test_pyarrow.pypass (493 passed, 1 skipped).Closes #4404