Skip to content

Reject colliding names in case-insensitive Schema lookups - #4389

Merged
Fokko merged 2 commits into
apache:mainfrom
rahulsmahadev:rahul/case-insensitive-name-collision
Nov 18, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
rahulsmahadev:rahul/case-insensitive-name-collision

Conversation

@rahulsmahadev

@rahulsmahadev rahulsmahadev commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

A case-insensitive lookup on a schema with two columns whose names differ only by case uses the last one. Schema._lazy_name_to_id_lower builds the lower-case index with a dict comprehension, so id overwrites ID, and Schema.find_field and Schema.select use that index for case-insensitive name lookups:

# columns ID (field 1) and id (field 2); rows (ID, id): (5, 7) and (6, 5)
tbl.scan(row_filter=EqualTo("ID", 5), case_sensitive=False).to_arrow()  # returns (6, 5), the row whose id is 5
tbl.delete("ID = 5", case_sensitive=False)  # removes (6, 5); (5, 7) is kept

The same wrong column is used by sort order updates and by schema updates (delete_column("ID") drops id, rename_column("ID", "key") renames id).

Java does not pick a column here. TypeUtil.indexByLowerCaseName throws Cannot build lower case index: ID and id collide, and Schema.caseInsensitiveFindField and SchemaUpdate use that index, so the lookup fails.

This change makes the index raise ValueError: Cannot build lower case index: ID and id collide (the same message as Java) when two different field IDs share a lower-case name. As in Java's Schema lookup, building the lower-case index fails for any requested name when the schema contains a collision, including when the requested name is unambiguous. Names that map to the same field ID, such as the short name of a list element struct field, are not collisions. Case-sensitive lookups do not change. StructType.field_by_name uses a separate lookup and is outside this change, so it remains a divergence from Java: it can still return the first matching field (it serves union_by_name). Java's Types.StructType.caseInsensitiveField builds its index with ImmutableMap.Builder.build(), which throws on duplicate keys, so a case-insensitive unionByNameWith on a struct with ID and id fails in Java. I will follow up with a separate PR for that.

Callers that treated any ValueError from a lookup as "not found" (UpdateSchema.add_column, the move methods, the identifier field check on commit, and the metadata column check in scan projection) now use a private lookup that returns None for a missing name and still raises for an ambiguous one. Without updating the add-column check, an index-only fix would let table.update_schema(case_sensitive=False).add_column("Id", LongType()).commit() add a third case variant.

Are these changes tested?

Yes. test_case_insensitive_operations_raise_on_case_variant_columns in tests/table/test_init.py runs a delete, a scan, a sort order update, and schema updates with case_sensitive=False on a table with ID and id, for the memory, sql and sql_without_rowcount catalogs. Each must raise, and the table must be unchanged afterwards. It fails on main, where the delete does not raise. test_case_sensitive_and_unambiguous_operations_are_unchanged checks exact-case lookups on the same table and a case-insensitive delete on a table without a collision; it passes on main and with this change. In tests/test_schema.py, test_schema_case_insensitive_lookup_of_case_variant_names_raises and test_schema_case_insensitive_lookup_of_nested_case_variant_names_raises fail on main with DID NOT RAISE, and test_schema_case_insensitive_lookup_of_list_element_struct_field covers the same-ID short names and passes on main and with this change. test_schema_case_insensitive_lookup_rejects_collision_for_any_name, test_metadata_column_projection_propagates_case_collision and test_identifier_field_lookup_propagates_case_collision cover unambiguous and missing names, the metadata column projection and the identifier field check; they fail on main.

Are there any user-facing changes?

Yes: case-insensitive name lookups through Schema.find_field and Schema.select now raise ValueError when the schema has names that differ only by case, instead of silently using the last column. Case-sensitive lookups are unchanged.

This pull request and its description were written by Isaac.

The lower-case name index kept only the last of two columns whose
names differ only by case (ID and id), so case-insensitive lookups
silently used it. Raise ValueError when two field IDs share a
lower-case name, as Java does, and let the schema update and metadata
column checks propagate that error instead of reading it as a missing
name. Exact-case lookups are unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, this updates the metadata-column lookup you added in #4375 so schema name collisions propagate. @kevinjqliu, you recently worked on Schema behavior in #3958. Could you take a look at the Java-compatible lookup behavior and the caller error handling when you have a chance?

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, the alignment with Java's TypeUtil.indexByLowerCaseName and Schema.caseInsensitiveFindField looks right to me, including failing for unambiguous names when the schema has a collision.

One correction to the description: it says StructType.field_by_name is outside this change and can still return the first matching field, which reads as if that matches Java. Java's Types.StructType.caseInsensitiveField builds its index with ImmutableMap.Builder.build(), which throws on duplicate keys, so a case-insensitive unionByNameWith on a struct with ID and id fails in Java. Could you reword that part to call it out as a remaining divergence? Fine to handle it in a follow-up issue.

Comment thread pyiceberg/schema.py
raise ValueError(f"Could not find field with id: {name_or_id}")
return self._lazy_id_to_field[name_or_id]

field = self._find_field_or_none(name_or_id, case_sensitive)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is the user facing change, could we also document the collision in the public docstrings? find_field and select still only say ValueError is raised when the field cannot be found. Something like ValueError: When case_sensitive is False and two fields have names that differ only by case. in both.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 6397704: find_field and select now list the collision under Raises.

Review feedback: the public docstrings now list the ValueError raised when
case_sensitive is False and two fields have names that differ only by case.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

Thanks @Fokko! I reworded the StructType.field_by_name paragraph to call it a remaining divergence from Java (caseInsensitiveField builds its index with ImmutableMap.Builder.build(), which throws on duplicate keys), and I'll follow up with a separate PR for it. The docstrings are updated in 6397704.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this @rahulsmahadev

@Fokko
Fokko added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit b104a9d Nov 18, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants