Repository navigation
Two concurrent create_namespace requests should raise a NamespaceAlreadyExistsError - #4408
Conversation
| if not properties: | ||
| properties = IcebergNamespaceProperties.NAMESPACE_MINIMAL_PROPERTIES |
There was a problem hiding this comment.
redundant with the next line
| if not properties: | ||
| properties = IcebergNamespaceProperties.NAMESPACE_MINIMAL_PROPERTIES | ||
| create_properties = properties if properties else IcebergNamespaceProperties.NAMESPACE_MINIMAL_PROPERTIES | ||
| namespace_string = Catalog.namespace_to_string(namespace, NoSuchNamespaceError) |
There was a problem hiding this comment.
avoid generating the namespace string in the for loop
Fokko
left a comment
There was a problem hiding this comment.
Thanks for the PR! Nice fix, this makes concurrent create_namespace behave like the other catalog operations instead of leaking a raw IntegrityError.
I pulled the branch and ran the tests:
tests/catalog/test_sql.pypasses, including the newtest_create_namespace_concurrent_integrity_error- Verified locally that the
Noneproperty path still raises the rawIntegrityError(not swallowed intoNamespaceAlreadyExistsError) and that a plain duplicate still raisesNamespaceAlreadyExistsError
Two small observations, feel free to ignore:
-
create_namespace(pyiceberg/catalog/sql.py, around line 617): theany(k is None or v is None ...)re-check is a heuristic to distinguish "bad input" from "concurrent create". It works becauseNonevalues fail the NOT NULL constraint, but a duplicate create with aNoneproperty value would now raiseIntegrityErrorinstead ofNamespaceAlreadyExistsError. That's an odd-input corner case, so I don't think it blocks, just something to be aware of. -
As you note in the PR body, two concurrent creates with disjoint property keys don't collide on the composite PK, so no error is raised and the namespace ends up with merged properties. Pre-existing behavior, but worth a follow-up issue if you want to close that gap.
CI looks green apart from markdown-link-check and python-integration-required, which look unrelated. LGTM once those are sorted.
Rationale for this change
Currently,
SqlCatalogcatches sqlalchemyIntegrityErrors on table operations (create_table,register_table,rename_table,commit_table) and converts them intoTableAlreadyExistsError. This PR updatescreate_namespaceto raise aNamespaceAlreadyExistsErrorif there are two concurrent requests to create the same namespace.Existing behaviour that throws an
IntegrityErrorif namespace property-value pairs containNone(as tested intests/catalog/test_catalog_behaviors.py) is preserved.Note that because the PK is a composite of catalog_name, namespace, and property_key, two concurrent requests with disjoint property keys do not trigger an
IntegrityError, so noNamespaceAlreadyExistsErrorwould be thrown.Are these changes tested?
Yes -- see
test_create_namespace_concurrent_integrity_errorintests/catalog/test_sql.py.Are there any user-facing changes?
No.