Skip to content

Two concurrent create_namespace requests should raise a NamespaceAlreadyExistsError - #4408

Merged
Fokko merged 3 commits into
apache:mainfrom
ialarmedalien:create_ns_race_condition
Nov 18, 2026
Merged

Fokko merged 3 commits into
apache:mainfrom
ialarmedalien:create_ns_race_condition

Conversation

@ialarmedalien

Copy link
Copy Markdown
Contributor

Rationale for this change

Currently, SqlCatalog catches sqlalchemy IntegrityErrors on table operations (create_table, register_table, rename_table, commit_table) and converts them into TableAlreadyExistsError. This PR updates create_namespace to raise a NamespaceAlreadyExistsError if there are two concurrent requests to create the same namespace.

Existing behaviour that throws an IntegrityError if namespace property-value pairs contain None (as tested in tests/catalog/test_catalog_behaviors.py) is preserved.

Note that because the PK is a composite of catalog_name, namespace, and property_key, two concurrent requests with disjoint property keys do not trigger an IntegrityError, so no NamespaceAlreadyExistsError would be thrown.

Are these changes tested?

Yes -- see test_create_namespace_concurrent_integrity_error in tests/catalog/test_sql.py.

Are there any user-facing changes?

No.

Comment thread pyiceberg/catalog/sql.py
Comment on lines -620 to -621
if not properties:
properties = IcebergNamespaceProperties.NAMESPACE_MINIMAL_PROPERTIES

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

redundant with the next line

Comment thread pyiceberg/catalog/sql.py
if not properties:
properties = IcebergNamespaceProperties.NAMESPACE_MINIMAL_PROPERTIES
create_properties = properties if properties else IcebergNamespaceProperties.NAMESPACE_MINIMAL_PROPERTIES
namespace_string = Catalog.namespace_to_string(namespace, NoSuchNamespaceError)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

avoid generating the namespace string in the for loop

@ialarmedalien ialarmedalien changed the title Two concurrent create_namespace requests raise a NamespaceAlreadyExistsError Two concurrent create_namespace requests should raise a NamespaceAlreadyExistsError Nov 17, 2026
@Fokko
Fokko self-requested a review November 17, 2026 17:29

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! Nice fix, this makes concurrent create_namespace behave like the other catalog operations instead of leaking a raw IntegrityError.

I pulled the branch and ran the tests:

  • tests/catalog/test_sql.py passes, including the new test_create_namespace_concurrent_integrity_error
  • Verified locally that the None property path still raises the raw IntegrityError (not swallowed into NamespaceAlreadyExistsError) and that a plain duplicate still raises NamespaceAlreadyExistsError

Two small observations, feel free to ignore:

  1. create_namespace (pyiceberg/catalog/sql.py, around line 617): the any(k is None or v is None ...) re-check is a heuristic to distinguish "bad input" from "concurrent create". It works because None values fail the NOT NULL constraint, but a duplicate create with a None property value would now raise IntegrityError instead of NamespaceAlreadyExistsError. That's an odd-input corner case, so I don't think it blocks, just something to be aware of.

  2. As you note in the PR body, two concurrent creates with disjoint property keys don't collide on the composite PK, so no error is raised and the namespace ends up with merged properties. Pre-existing behavior, but worth a follow-up issue if you want to close that gap.

CI looks green apart from markdown-link-check and python-integration-required, which look unrelated. LGTM once those are sorted.

@Fokko
Fokko added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit ba3ae08 Nov 18, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants