Skip to content

Read deletion vectors whose first bitmaps are empty - #4425

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/deletion-vector-high-positions
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/deletion-vector-high-positions

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Bug: DeletionVector.to_vector() builds the PyArrow array without a type, so PyArrow infers it from the first bitmap. A deletion vector has one bitmap per high 32 bits of the position. When all deleted positions are at or above 232 (a data file with more than 232 rows), the first bitmap is empty, PyArrow types it as null and the read fails:

from pyroaring import BitMap
from pyiceberg.table.deletion_vector import DeletionVector

DeletionVector("data.parquet", [BitMap(), BitMap([5])]).to_vector()
# pyarrow.lib.ArrowInvalid: Invalid null value

A vector with no bitmaps, like 64mapempty.bin in tests/table/bitmaps, fails with ArrowInvalid: cannot construct ChunkedArray from empty vector and omitted type. Java writes an empty bitmap for key 0 in the first case (RoaringPositionBitmap.serialize) and reads both fine.

Fix: pass type=pa.int64(). This also covers from_positions() in #3474, which pads missing keys with empty bitmaps the same way.

Are these changes tested?

Yes, added test_to_vector_with_empty_bitmaps_for_low_keys and extended test_map_empty in tests/table/test_deletion_vector.py. Both fail on main with the errors above and pass with this change.

Are there any user-facing changes?

Yes, these deletion vectors can be read now. A vector with only empty bitmaps now converts to an empty int64 array instead of a null-typed one.

Please add the changelog label.

This pull request and its description were written by Isaac.

DeletionVector.to_vector() built its chunked array without a type, so an
empty first bitmap (positions only at or above 2**32) or a vector with no
bitmaps raised ArrowInvalid, and a vector of only empty bitmaps came back
null-typed. Build it as int64.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@kevinjqliu, this fixes DeletionVector.to_vector() for vectors whose first bitmaps are empty, in the read path you reviewed in #1516. @ArslanAnjum, you last changed deletion_vector.py in #4181. Could you take a look when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant