Skip to content

Row lineage: Read _last_updated_sequence_number - #4413

Open
rambleraptor wants to merge 2 commits into
apache:mainfrom
rambleraptor:row-lineage-read-seq-number
Open

rambleraptor wants to merge 2 commits into
apache:mainfrom
rambleraptor:row-lineage-read-seq-number

Conversation

@rambleraptor

Copy link
Copy Markdown
Collaborator

Rationale for this change

I can break this up into a couple separate PRs, but I wanted to finish up row lineage.

This does two things:

  • Puts first_row_id in inspect.files()
  • Lets users read _last_updated_sequence_number

Are these changes tested?

Included unit test + updated our Spark test (from the last PR)

Are there any user-facing changes?

@rahulsmahadev rahulsmahadev left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rambleraptor! Inheriting from the data sequence number, and the nulls for files without a first_row_id, match the spec. I left one suggestion inline about REST-planned scans.

A couple of smaller suggestions:

  • The new first_row_id: int64 line in api.md is in the inspect.entries() example; I think it belongs with files.
  • Since this adds first_row_id to inspect.files() and makes _last_updated_sequence_number readable, it may be worth adding the changelog label.

Comment thread pyiceberg/io/pyarrow.py Resolved
Carry the data sequence number of each data file on FileScanTask, and fill
null _last_updated_sequence_number values with it, as the Java readers do.
The value is null when the file has no first row ID or no data sequence
number, such as for files planned by a REST server.

Part of apache#4080.
Add the first_row_id column to the files metadata tables, matching the
Java files table schema.

Part of apache#4080.
@rambleraptor
rambleraptor force-pushed the row-lineage-read-seq-number branch from f823088 to 0cdeab5 Compare November 18, 2026 21:20
@rambleraptor

Copy link
Copy Markdown
Collaborator Author

@rahulsmahadev thanks so much for your review!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants