Skip to content

Raise on case-insensitive duplicate names in StructType.field_by_name - #4424

Open
rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/struct-case-insensitive-duplicates
Open

rahulsmahadev wants to merge 1 commit into
apache:mainfrom
rahulsmahadev:rahul/struct-case-insensitive-duplicates

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

Follow-up to #4389.

Bug: StructType.field_by_name(name, case_sensitive=False) returns the first match when a struct has two fields that differ only by case, like ID and id. So a case-insensitive union_by_name can silently pick the wrong column:

table = catalog.create_table("default.t", Schema(NestedField(1, "ID", LongType()), NestedField(2, "id", LongType())))
with table.update_schema(case_sensitive=False) as update:
    update.union_by_name(Schema(NestedField(1, "id", LongType())))

Fix: raise ValueError: Cannot build lower case index: ID and id collide, the same message as #4389. Java fails here too: Types.StructType.caseInsensitiveField builds its index with ImmutableMap.Builder, which throws on duplicate keys.

Remaining differences from Java:

  • Java caches the lower-case index per struct. This PR rebuilds it on every lookup, which is slower on very wide schemas. Caching can be a follow-up.
  • Java also validates the whole schema at the end of a schema update (SchemaUpdate.applyChanges). PyIceberg doesn't, so for example a union that adds a new struct with both ID and id still succeeds.

Are these changes tested?

Yes, added test_struct_field_by_name_case_insensitive_duplicates in tests/test_types.py and test_union_by_name_case_insensitive_duplicates in tests/catalog/test_catalog_behaviors.py, which runs on the memory, sql and sql_without_rowcount catalogs. Both fail on main with DID NOT RAISE ValueError and pass with this change.

Are there any user-facing changes?

Yes, case-insensitive lookups on such a struct now raise ValueError instead of returning the first field, and so does a case-insensitive union_by_name that looks up a field in it. Case-sensitive lookups don't change.

Please add the changelog label.

This pull request and its description were written by Isaac.

field_by_name(case_sensitive=False) returned the first field on a struct with ID and id, so
union_by_name silently matched a new id column to ID. Raise ValueError on any case-insensitive
lookup of such a struct, as Java's caseInsensitiveField does, so union_by_name fails instead.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@Fokko, this is the follow-up for the StructType.field_by_name divergence you pointed out in #4389. Could you take a look when you have a chance?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant