Repository navigation
Read deletion vectors whose first bitmaps are empty - #4425
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
DeletionVector.to_vector() built its chunked array without a type, so an empty first bitmap (positions only at or above 2**32) or a vector with no bitmaps raised ArrowInvalid, and a vector of only empty bitmaps came back null-typed. Build it as int64. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@kevinjqliu, this fixes |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Bug:
DeletionVector.to_vector()builds the PyArrow array without a type, so PyArrow infers it from the first bitmap. A deletion vector has one bitmap per high 32 bits of the position. When all deleted positions are at or above 232 (a data file with more than 232 rows), the first bitmap is empty, PyArrow types it as null and the read fails:A vector with no bitmaps, like
64mapempty.binintests/table/bitmaps, fails withArrowInvalid: cannot construct ChunkedArray from empty vector and omitted type. Java writes an empty bitmap for key 0 in the first case (RoaringPositionBitmap.serialize) and reads both fine.Fix: pass
type=pa.int64(). This also coversfrom_positions()in #3474, which pads missing keys with empty bitmaps the same way.Are these changes tested?
Yes, added
test_to_vector_with_empty_bitmaps_for_low_keysand extendedtest_map_emptyintests/table/test_deletion_vector.py. Both fail on main with the errors above and pass with this change.Are there any user-facing changes?
Yes, these deletion vectors can be read now. A vector with only empty bitmaps now converts to an empty int64 array instead of a null-typed one.
Please add the
changeloglabel.This pull request and its description were written by Isaac.