Skip to content

Data streams v2: conformance tests ported from the Rust crate, plus remaining API - #325

Open
1egoman wants to merge 11 commits into
mainfrom
data-streams-v2-conformance-tests
Open

1egoman wants to merge 11 commits into
mainfrom
data-streams-v2-conformance-tests

Conversation

@1egoman

@1egoman 1egoman commented Nov 22, 2026

Copy link
Copy Markdown

Ports the unit tests of the Rust livekit-data-stream crate to end-to-end C++ tests, so we can show the C++ SDK sends and receives data streams the same way every other SDK does, including data streams v2 (inline and compressed streams). It also adds the two remaining API pieces the tests exposed.

This branch contains the commits from #321 and #322 (cherry-picked), so it can land instead of them, or after them.

How the branch is ordered

  1. Harness and receive-side tests (806d460..c9aead9). These use only API that exists on main. Against main, 22 of the 27 fail:
    • inline v2 streams are never delivered;
    • compressed chunks reach the app as raw deflate bytes;
    • duplicate chunks are appended;
    • length, gap and close-reason errors are never surfaced;
    • streams stay open when a participant disconnects.
  2. Data streams: receive via FFI readers; add sendText / sendBytes / sendFile #321 and Data streams: port writers to FFI writer handles; add streamText / streamBytes #322, cherry-picked. All 27 pass after these.
  3. Send-side tests, then the receive payload cap and reader->close() tests. These need the new API, so they come after the picks.
  4. New API, each commit with its tests:
    • TextStreamOperationType plus operation_type / version / generated on StreamTextOptions and TextStreamInfo. Agents set generated on transcriptions.
    • ByteStreamReader::writeToFile(). This wraps the FFI call; Rust rejects sender-chosen names that aren't plain file names.
  5. Mapping doc (src/tests/integration/data_stream_conformance.md). It lists every one of the 124 Rust tests as ported, partially ported, or N/A with the reason.

How the tests work

  • FfiPeer drives the other participant through the raw FFI, so it sends exactly what Python, Unity and other FFI-based SDKs send. Its one-shot sends go through the Rust outgoing manager; its raw requests send hand-built v1 packets.
  • RawPacketSniffer records the raw packets a room receives. The raw header event doesn't carry inline_content or compression, so:
    • "inline" means a header arrived with no chunk or trailer packets;
    • "compressed" means the chunk bytes differ from the payload and are smaller than it.
  • Collectors receive through the public C++ API.
  • scripts/set-test-tokens.sh now also mints an optional LIVEKIT_TOKEN_C, for tests that need a third participant. Those tests skip without it.

What can't be tested from C++

Everything below is delegated to the Rust crate, whose own tests cover it.

  • Progress: the FFI exposes no progress API.
  • Pre-v2 recipients and recipients without compression: every FFI client advertises v2 with compression, so a test can't create one.
  • Malformed v2 packets: the FFI can only hand-build v1 packets.
  • Per-packet encryption mismatches.
  • Manager internals: stream counts, StreamClosed, batching.
  • The outgoing sender_identity stamp: the SFU re-attributes packets unless the token allows impersonation.

Behaviour change in writeToFile

Write-to-file and incremental reading both consume the FFI reader handle, so byte readers now start reading lazily:

  • the first readNext() / readAll() starts reading, or writeToFile() hands the stream to the SDK core;
  • a reader that is closed or dropped before either drops its handle;
  • Room now tracks byte readers by weak_ptr, so a reader the app ignores releases its stream.

Text readers are unchanged.

TextStreamInfo gains members, so code built against older headers must be recompiled.

Verification (local, macOS arm64, livekit-server --dev)

  • Unit tests: 384/384.
  • Data stream integration tests: all 107 pass.
  • Full integration suite: 173 pass. The 4 failures (three DataTrack schema tests and ServerDeletedRoomDisconnectsAndTearsDownLocally) also fail on main on this machine.
  • One flaky test: SenderDisconnectOnlyAffectsItsOwnStreams failed once during a burst of local ICE errors. It then passed 5/5 on repeat, and I gave its final wait more time.
  • Python livekit 1.3.0, both directions: Python → C++ intact 3/3, C++ → Python intact 2/2.
  • Tooling:

Windows and Linux are untested until CI runs.

🤖 Generated with Claude Code

1egoman and others added 11 commits November 23, 2026 08:52
Shared helpers for porting the Rust livekit-data-stream crate's tests to
end-to-end C++ tests:

- FfiPeer drives a room's participant through the raw FFI, so it sends
  exactly what another FFI-based SDK sends: one-shot sends and stream
  writers go through the Rust outgoing manager (data streams v2), and raw
  header/chunk/trailer requests send hand-built v1 packets.
- RawPacketSniffer records the raw stream packets a room receives.
- Text/ByteStreamCollector receive through the public C++ API and record
  content, info and errors.

The harness only uses API that predates the FFI reader/writer port, so
tests built on it compile before and after that port.

Also mint an optional LIVEKIT_TOKEN_C (cpp-test-c) for tests that need a
third participant, and add RoomTestAccess::roomHandleId.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the incoming-manager tests that a v1 (multi-packet, uncompressed)
sender can reproduce: round trips, trailer attribute merging, too few /
too many bytes against total_length, duplicate and empty chunks, chunk
gaps, trailer reasons, orphan packets, remote stream writers, and sender
or receiver disconnects mid-stream.

On main, 12 of these 17 tests fail: the C++ receive path appends
duplicate chunks, never surfaces length/gap/reason errors, and leaves
streams open when a participant disconnects.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the incoming-manager tests for data streams v2 as sent by other
SDKs: inline streams (compressed and uncompressed, empty, large but highly
compressible), multi-packet compressed text and byte streams, UTF-8
reframing of decompressed text, and text streams with attachments.

All 10 fail on main: inline streams are never delivered and compressed
chunks are delivered as raw deflate bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sendFile

Incoming data streams were rebuilt from the low-level stream_header/chunk/
trailer events. Since livekit-ffi 0.15.0 always advertises data streams v2,
modern senders send C++ participants single-packet and deflate-compressed
streams, which that path can't read: inline streams hang readAll() forever
and compressed chunks arrive as raw deflate bytes. The FFI also handed C++ an
owned reader handle for every incoming stream, which C++ ignored, so every
stream's content stayed buffered in the FFI for the life of the process.

Receive:
- Room consumes text/byte_stream_opened: unhandled topics drop the reader
  handle; handled topics start read_incremental and route the resulting
  *_stream_reader_events (keyed by reader handle) to the reader. The
  low-level stream_*_received events are now ignored.
- Readers surface abnormal ends (sender close reason, size limit, disconnect)
  as DataStreamError after draining buffered chunks, merge end-of-stream
  attributes into info() on the reading thread, gain close(), and
  ByteStreamReader gains readAll(). Room shutdown wakes blocked readers.
- RoomOptions::data_stream.max_payload_byte_length sets the receive limit.

Send:
- LocalParticipant::sendText / sendBytes / sendFile use the FFI one-shot
  sends (compression and single-packet delivery when recipients support
  them). They throw DataStreamError on failure and refuse to reach the FFI
  once the room has disconnected, since the FFI panics there.

TextStreamOpenedEvent / ByteStreamOpenedEvent::reader_handle stays 0 and is
documented as deprecated. Existing writers are unchanged (v1 framing is still
accepted by FFI readers).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kChatMessage and the low-level stream packet events both just break, which
bugprone-branch-clone flags as consecutive identical branches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Bytes

TextStreamWriter / ByteStreamWriter now open an FFI writer
(text/byte_stream_open) and send through text/byte_stream_write and
*_stream_close instead of hand-built header/chunk/trailer packets. Rust does
the chunking and UTF-8 splitting.

- LocalParticipant::streamText / streamBytes return an already-open writer
  (blocking until the open callback, throwing DataStreamError on failure).
  The public writer constructors keep working but are deprecated; they open
  lazily on the first write or close, as the header did before.
- write() and close() share one mutex and each waits for its FFI callback,
  so a close never overtakes an in-flight write (python-sdks#879); this also
  fixes close() not taking the write lock.
- A LocalParticipant tracks its writers in a registry shared with them. On
  disconnect (or participant destruction) every writer is closed and its
  handle dropped, so later write/close throw DataStreamError instead of
  reaching an FFI session that would panic.
- A writer destroyed without close() sends a non-waiting FFI close. Rust's
  writer only sends its drop-time trailer from a Tokio thread, so a plain
  handle drop from C++ left receivers waiting forever.
- kStreamChunkSize and the protected ensureHeaderSent / sendChunk /
  sendTrailer helpers are deprecated and reimplemented on the new path.
- Removed FfiClient::sendStream*Async, splitUtf8 and the now-unused
  header/chunk/trailer converters.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the outgoing-manager tests that are observable end to end: the C++
SDK sends through sendText / sendBytes / sendFile / streamText /
streamBytes, and the receiving room's raw packets and reader output are
checked. Covers inline vs multi-packet decisions, compression (one-shot
and sendFile), compress=false, MTU splitting (including UTF-8 aware
text splits), attachments, empty payloads, writer trailers (reason and
attributes), header size limits, writer drop, targeting, and sender
identity attribution.

The raw header event does not carry inline_content or compression, so
inline is observed as a header with no chunk/trailer packets and
compression as chunk bytes that differ from and are smaller than the
payload. The SFU attributes packets to the authenticated participant
for standard tokens, so the sender_identity tests check acceptance and
delivery rather than the outgoing stamp.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ports the incoming-manager max_payload_byte_length tests (unknown total,
declared total fast-fail, exactly-at-cap, inline and compressed
breaches) through RoomOptions::data_stream, and the closer tests through
TextStreamReader::close().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Other SDKs set these text header fields (agents mark transcriptions with
generated=true), but the C++ API could neither send nor see them. Add
TextStreamOperationType plus operation_type / version / generated to
StreamTextOptions and TextStreamInfo, and carry them through one-shot
sends, stream writers and received streams. Defaults are left off the
wire, matching what the SDK core treats as unset.

TextStreamInfo gains members, so code built against older headers must
be recompiled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wrap the FFI's write-to-file so a received byte stream can be written
straight to disk by the SDK core, which also rejects sender-chosen file
names that are not plain file names (path separators, "..", absolute
paths, empty).

Write-to-file and incremental reading both consume the FFI reader
handle, so byte readers now start reading lazily: the first readNext() /
readAll() starts incremental reading, writeToFile() hands the stream to
the core instead, and a reader closed or destroyed unconsumed drops its
handle. Room tracks byte readers weakly so an ignored reader releases
its stream rather than being kept alive by the room.

Ports the Rust write_to_file tests end to end, plus the one-consumer
rule, the event-thread guard and ignored readers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Map every test of the Rust livekit-data-stream crate to its C++
conformance test, or say why it cannot be expressed from the C++ layer
(no progress API, no pre-v2 recipients, malformed v2 packets,
per-packet encryption, manager internals, SFU sender attribution).

Also document writeToFile and lazy byte reading in README / AGENTS.md,
and give the three-participant disconnect test more time for its final
stream, which can be slow to drain on a congested local network.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@1egoman
1egoman marked this pull request as ready for review November 22, 2026 21:17
@1egoman
1egoman requested review from a team and xianshijing-lk as code owners November 22, 2026 21:17
Comment on lines +1 to +8
# Data stream conformance tests

The `test_data_stream_conformance_*.cpp` integration tests port the unit tests
of the Rust `livekit-data-stream` crate
(`client-sdk-rust/livekit-data-stream/src`) to end-to-end C++ tests against a
real SFU. They check that the C++ SDK sends and receives data streams the way
every other LiveKit SDK does, including data streams v2 (inline single-packet
streams and deflate-compressed streams).

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: drop this file before merging.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 6 potential issues.

6 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread src/local_participant.cpp
Comment on lines +103 to +107
// The FFI panics (and the SDK raises SIGTERM) if a stream is sent after the
// room session is gone, so refuse here instead.
if (disconnected_.load(std::memory_order_acquire)) {
throw DataStreamError(std::string("LocalParticipant::") + operation + ": not connected");
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Process termination during data stream sends

When a send overlaps disconnect, dataStreamHandle can approve it without keeping the room session alive. Rust unwraps the expired session, and ffiEventCallback terminates the process instead of returning an error.

Learn more

A retained LocalParticipant keeps the participant handle alive, not the room session. Room shutdown detaches the room handle and marks the participant disconnected, then releases the room handle at the end of shutdown. Passing the check does not pin that handle while sendBytesAsync builds its request or while the Rust task waits to run. The pinned Rust SDK's LocalParticipant::send_bytes, send_text, send_file, stream_text, and stream_bytes call session().unwrap(), where session() upgrades a weak reference. The same gap applies to opening a writer after participantHandle returns. Rust forwards task panics to ffiEventCallback, which raises SIGTERM.

Example: One thread begins sending a large byte buffer and passes the connection check. Another thread disconnects the room while the buffer is being copied into the FFI request. The sender still holds the participant, but the Rust task no longer finds the room session and panics.

Recommended fix: Make the pinned Rust data-stream entry points return a typed not-connected error when they cannot upgrade the weak session reference. Keep the C++ checks as fast paths, and add coverage for a send crossing room teardown.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread include/livekit/room.h
Comment on lines +139 to +140
/// Data stream settings, such as the incoming stream size limit.
DataStreamOptions data_stream;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Binary incompatibility on room connect

Prebuilt clients pass the old RoomOptions layout when loading the updated SDK. toProto reads other_sdks at its new offset, outside the old field and potentially outside the object.

Learn more

Room::connect takes RoomOptions by reference, and its exported signature has not changed. A client compiled with the old header still creates the old object, even when it connects using default settings. The new layout puts a receive limit before other_sdks, so the old other_sdks storage is interpreted as the new limit while toProto reads other_sdks from the new offset. The repository public API rules forbid breaking existing consumers. Changes to exported stream types, including TextStreamInfo, also require an ABI review.

Example: An application built against the prior SDK calls Room::connect with a default RoomOptions after only its shared library is updated. The updated converter reads the optional string's presence flag beyond the old object.

Recommended fix: Preserve the existing layout and overload. Expose the new limit through a versioned options type or a separate entry point that does not reinterpret existing callers' storage. Check with a consumer compiled against the base headers and run against the new library.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/data_stream.cpp
Comment on lines +281 to +284
try {
std::future<std::string> written;
try {
written = FfiClient::instance().byteStreamWriteToFileAsync(*handle, directory, name_override);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Failed downloads overwrite existing files

writeToFile truncates an existing destination before the incoming stream completes. If the sender aborts or disconnects, the call throws and leaves an empty or partial replacement.

Learn more

byteStreamWriteToFileAsync forwards the destination to the pinned Rust SDK's ByteStreamReader::write_to_file (livekit-data-stream/src/incoming/stream_reader.rs:153-181). Rust creates or truncates the final path before consuming the stream. If a read fails, it returns immediately, leaving whatever bytes were written in place. The C++ error path reports the failure but cannot recover the old file.

Example: A client has a complete report.pdf on disk. It saves an updated report.pdf, but the sender aborts after one chunk. The call throws, and the original report has already been replaced with a fragment.

Recommended fix: Stage the incoming bytes in an exclusive temporary file in the destination directory. Replace the final path only after the stream and file writes succeed, and clean up the staging file on failure. Add an interrupted-stream test that verifies an existing destination remains unchanged.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/room.cpp
Comment on lines +1428 to +1431
for (const auto& kv : byte_readers) {
if (const auto reader = kv.second.lock()) {
reader->onStreamEnded({}, std::string(kDisconnectedWhileReceiving));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Disconnect loses completed byte streams

When a fully received byte stream has not been read, endStreamReaders marks it failed on disconnect. A later readAll throws instead of returning the bytes still buffered in Rust.

Learn more

onByteStreamOpened gives the app a reader immediately, but leaves all data in Rust until the app claims the handle. Completed transfers do not generate C++ reader events before startReading, so even completed transfers remain in the room's reader map. Room shutdown takes away that map and sets a terminal error on every reader in it. The unread handle still exists, but its later chunk events have no route and the terminal error cannot be replaced. The old byte handler buffered incoming chunks before the first application read; a retained reader whose trailer had arrived remained usable after disconnect.

Example: A chat app retains an attachment reader, the sender finishes transferring an image, and the user leaves the room. Opening the attachment from chat history calls readAll() and gets a disconnect error although the entire image had arrived.

Recommended fix: Keep event routing for claimed byte readers independent of the room listener's lifetime, and preserve unread handles for retained readers. Let the core distinguish completed transfers from interrupted ones instead of assigning both the same terminal error. Test reading a completed, unconsumed byte stream after disconnect.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/local_participant.cpp

ByteStreamInfo LocalParticipant::sendFile(const std::string& path, const StreamByteOptions& options) {
const auto handle_id = dataStreamHandle("sendFile");
proto::StreamByteOptions proto_options = toProto(options);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 File sends lose MIME type

When sendFile receives no MIME override, Rust sends application/octet-stream regardless of the filename. Receivers that dispatch by MIME type cannot recognize files such as PNG images.

Learn more

StreamByteOptions documents file-specific MIME inference when the MIME type is unset. toProto leaves that field unset, and the pinned Rust SDK's send_file calls build_byte_header with that missing value (livekit-data-stream/src/outgoing/manager.rs:308-339,507-529). The builder always falls back to the generic byte-stream MIME type. The existing SendFileRoundTrip checks only that the type is nonempty and matches on both sides.

Example: Sending photo.png with default options gives receivers application/octet-stream instead of image/png, so an image handler keyed by MIME type does not run.

Recommended fix: Implement file MIME inference when no override is present, preferably in the Rust file-send path so the SDKs agree. Preserve explicit overrides and test the metadata for a known file type.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread src/data_stream.cpp
try {
std::future<std::string> written;
try {
written = FfiClient::instance().byteStreamWriteToFileAsync(*handle, directory, name_override);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟥 Stream downloads follow destination symlinks

When writeToFile uses a sender-chosen name matching an existing symlink, the core follows the link. The download can overwrite a file outside the receiver’s selected directory.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant