Skip to content

fix(io): isolate S3FileSystem per signed FsspecFileIO and re-register… - #4418

Open
SurenderguliaEnginner wants to merge 1 commit into
apache:mainfrom
SurenderguliaEnginner:fix/fsspec-s3-signer-isolation
Open

SurenderguliaEnginner wants to merge 1 commit into
apache:mainfrom
SurenderguliaEnginner:fix/fsspec-s3-signer-isolation

Conversation

@SurenderguliaEnginner

Copy link
Copy Markdown

Closes #4414

Summary

  • When s3.signer is configured in pyiceberg.io.fsspec._s3, instantiate a private _SignedS3FileSystem(S3FileSystem) subclass with skip_instance_cache=True so signed FsspecFileIO instances never share a cached S3FileSystem or race on fs.s3.meta.events.unregister / register_last.
  • Override async def set_session on _SignedS3FileSystem to re-register the signer events if s3fs recreates the underlying aiobotocore S3 client.
  • Register the signer on the initial fs.s3 client before returning fs.
  • Added unit tests in tests/io/test_fsspec.py:
    • test_s3_signer_isolated_across_file_ios
    • test_s3_signer_re_registered_on_set_session

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FsspecFileIO sends unsigned S3 requests: signer is swapped on a shared, cached S3FileSystem

1 participant