Repository navigation
Raise on case-insensitive duplicate names in StructType.field_by_name - #4424
Open
rahulsmahadev wants to merge 1 commit into
Open
rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
field_by_name(case_sensitive=False) returned the first field on a struct with ID and id, so union_by_name silently matched a new id column to ID. Raise ValueError on any case-insensitive lookup of such a struct, as Java's caseInsensitiveField does, so union_by_name fails instead. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Follow-up to #4389.
Bug:
StructType.field_by_name(name, case_sensitive=False)returns the first match when a struct has two fields that differ only by case, likeIDandid. So a case-insensitiveunion_by_namecan silently pick the wrong column:Fix: raise
ValueError: Cannot build lower case index: ID and id collide, the same message as #4389. Java fails here too:Types.StructType.caseInsensitiveFieldbuilds its index withImmutableMap.Builder, which throws on duplicate keys.Remaining differences from Java:
SchemaUpdate.applyChanges). PyIceberg doesn't, so for example a union that adds a new struct with bothIDandidstill succeeds.Are these changes tested?
Yes, added
test_struct_field_by_name_case_insensitive_duplicatesintests/test_types.pyandtest_union_by_name_case_insensitive_duplicatesintests/catalog/test_catalog_behaviors.py, which runs on the memory, sql and sql_without_rowcount catalogs. Both fail on main withDID NOT RAISE ValueErrorand pass with this change.Are there any user-facing changes?
Yes, case-insensitive lookups on such a struct now raise
ValueErrorinstead of returning the first field, and so does a case-insensitiveunion_by_namethat looks up a field in it. Case-sensitive lookups don't change.Please add the
changeloglabel.This pull request and its description were written by Isaac.