Skip to content

Parse protobuf battle journals and keep the JSON journal shape - #392

Merged
netniV merged 1 commit into
STFC-Mod:devfrom
blevouschurverantunt:fix/journal-protobuf
Nov 22, 2026
Merged

netniV merged 1 commit into
STFC-Mod:devfrom
blevouschurverantunt:fix/journal-protobuf

Conversation

@blevouschurverantunt

@blevouschurverantunt blevouschurverantunt commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

What changed and why

Since 2026-11-19 the game server answers POST /journals/get with protobuf instead of JSON. The body is a
ServiceResponse whose entity group 221 (TYPE_BATTLEREPORT) holds a JournalResponse. FetchBattle still parsed
the body as JSON, so every journal failed and battle log sync stopped for all targets. Fixes #391.

The mod now converts the JournalResponse back into the journal object the JSON server sent. It keeps the same key
names, nesting and JSON types, so sync consumers and the battlelog record stay unchanged. Bodies whose first
non-whitespace byte is { still take the JSON path.

  • src/mod/sync/journal.h, journal.cc (new): IsJsonBody() and JournalFromProtobuf(). They depend only on the
    generated protobuf code, nlohmann::json and the standard library. Conversion rules:
    • int64 ids stay JSON integers, all digits.
    • float32 values go out in shortest form (1.1, not 1.100000023841858), through to_chars and the JSON parser,
      as ShortNumber does.
    • Map keys become decimal strings, and battle_time is YYYY-MM-DDTHH:MM:SS UTC.
    • battle_log entries keep their number text, so 5.0 stays a float.
    • bridge_officers is the 10-slot crew array with nulls, {} for an NPC side and null for an armada.
    • Malformed, truncated or oversized bodies are refused without throwing.
  • src/mod/sync/battlelogs.cc: FetchBattle picks the path from the first byte and logs one error with the battle
    id, body size and reason when a protobuf journal does not convert. As before, it then retries. Participants now
    reads deployed_fleets as an object keyed by fleet id, as journals have always sent it. The old is_array()
    check skipped it, so only initiator_id/target_id were resolved. Empty uids are skipped.
  • src/mod/game/proto/Digit.PrimeServer.Models.proto: 10 new messages (JournalResponse, JournalFleetData,
    JournalFleetOfficerEntry, BridgeCrewOfficer, CargoProto, JournalChestDrop, JournalContextData,
    JournalContextDataInner, JournalMultitargetArmadaInfo, JournalGalacticAnomalyData). They were recovered from
    the client's message descriptors and checked against live responses. These are pure additions: existing messages,
    including DeployedFleet and JournalHeaderProto, are untouched.
  • zig/deps/protobuf_roots.txt: JournalResponse and ServiceResponse.
  • tests/sync_logic.cc, zig/tests.zig: a protobuf journals test in test-sync.
  • CONTRIBUTING.md: sync/journal.cc in the sync ownership table.

Converted journals carry "journal_src":"pb1" inside the journal object, so a consumer can tell them apart.

Keys the server no longer sends are left out, never set to 0 or invented. They return automatically if the server
sends the backing field again:

  • Journal: num_references.
  • Chest drop: loot_roll_key, loot_roll_type.
  • Chests gained, params: chest_name, items, chest_item_<id>_amount/_type, source, id.
  • Fleet data (both sides): faction_id, active_officer_slots; and fleet_id on an armada side, whose old value is
    not derivable.
  • Deployed fleet:
    • battle_start_time, state, is_active and the other is_* flags;
    • node_id, course_id, latest_course_vector_x/_y, state_updated_at;
    • internal_stats, mining_amount_bonus_stats, node_address;
    • active_buffs[].attributes.

xp_gained and warp_data stay null, as in every JSON journal. Values that were server doubles are now float32
on the wire. Integral values are exact; fractional stats and ratings above 2^24 can differ in the last digits.

Validation

  • Run locally on Windows with Zig 0.16: zig build mods build-tests -Dtarget=x86_64-windows-msvc,
    zig build test-sync test-mod test-codegen, the full ReleaseFast DLL build, and zig build fmt.
  • The new protobuf journals test builds a synthetic player-vs-NPC ServiceResponse with the protobuf API. It
    compares the result with an exact expected journal, number kinds included. It also checks that a JSON body is
    detected, and that empty, garbage, truncated, wrong-group and id-less bodies are refused.
  • Parity: the converter's output for 20 captured journals (battle types 2 and 5) matches a separate reference decoder,
    integers digit for digit and all keys and types. The only differences are float32 values that sit exactly halfway
    between two shortest forms, where to_chars rounds half to even. A value-level self-check of 1,430 assertions
    against the decoded wire passed.
  • In game (the same converter, ported to the 1.1.10 release layout): one session of about four minutes sent 84 battle logs to a private sync endpoint, all hostile fights (battle type 5), 83 wins and one loss. All 84 were accepted and parsed by the consumer with the same columns as journals from before the server change, apart from what was derived from the omitted keys. The mod log had no errors, where sessions before the fix logged a parse error for every battle.
    The dev build is compile-, unit- and parity-tested only, not run in game.
  • CI: build-win currently fails on the runner's newer MSVC toolset, with STL1000 in every unit including
    dependencies. This is unrelated to this change.

Settings and platforms

  • No new or changed settings. Battle fetching stays opt-in (battlelogs).
  • Untested battle types: multi-participant armadas, PvP, station and base raids, hazards and wave defense. They follow
    the same rules. A single-ship armada (type 8) was validated independently, see the comments. An armada
    keeps deployed_fleets keyed by fleet id, bridge_officers is null and the side's fleet_id is
    left out. A side without a fleet leaves out deployed_fleet.
  • For sync-site maintainers: the record shape is unchanged apart from the omitted keys above and the journal_src
    marker. Match armada participants by hull or ship id, not by array position. Measurement fields can differ in
    number kind from the JSON journals: ratings, hit points and shields are int64 on the wire, coords and the
    *_level_percentages are float32. Ids and battle_log markers stay integers.
  • Not built or run on macOS locally. The converter avoids float from_chars, which Apple's libc++ lacks
    before macOS 26.

🤖 Generated with Claude Code

Since 2026-11-19 the game server answers POST /journals/get with protobuf
instead of JSON: a ServiceResponse whose entity group 221 (BattleReport)
holds a JournalResponse. The JSON parse failed, so no battle journal
reached any sync target.

The mod now converts the JournalResponse into the journal object the JSON
server sent: same keys, nesting and JSON types, so sync consumers keep
working unchanged. Keys the server no longer sends are left out rather
than invented, and converted journals carry "journal_src":"pb1". Bodies
that start with '{' still take the JSON path.

The battle report messages are added to the models proto as new messages
only; existing messages are untouched. Participant lookup now reads
deployed_fleets as the object it is in journals.

Validated with a synthetic unit test and against 20 captured journals of
single-fleet battles. Armada, PvP and station battles are untested.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@kcid9660

Copy link
Copy Markdown
Contributor

Independent validation from a second setup. I was chasing the same break when this landed, so I checked it against what I had. It works for me; there are two small notes below.

Wire and schema

  • I captured a live POST /journals/get response from the current client (1.000.53284; serverVersion reports m96), a type 2 battle against a Marauder. It is exactly as described: ServiceResponse → entity group 221 → JournalResponse.

  • I generated a proto for the whole JournalResponse type closure straight from an Il2CppDumper dump of that client (34 messages). It decodes the capture with no unknown fields. Against your 10 new messages, all 68 field slots agree on number, label and message type. The only differences are two integer widths where the client declares int:

    • BridgeCrewOfficer.slotIndex: int in the client (int32); the PR has int64.
    • JournalGalacticAnomalyData.systemBuffs values: IDictionary<long, int> (map<int64, int32>); the PR has int64.

    Both are varints and negative int32s are sign-extended on the wire, so decoding is identical. I'm only mentioning them for fidelity with the client's descriptors.

Converter output

  • JournalFromProtobuf converts the capture (56,324 bytes in, 200,655 bytes of JSON out). Ids are digit-exact and battle_time has the old format.
  • A structural diff against a pre-change JSON journal of the same battle type (2026-11-15) shows only the documented omissions plus journal_src, with one exception: state_updated_at is missing from deployed_fleet and deployed_fleets entries, and it isn't in the "keys left out" list. journal.cc never references it (it is DeployedFleet field 51), so I think it just belongs in that list. The consumer I tested doesn't use it.
  • "Same JSON types" holds at the JSON level, but the integer/float kind differs from the old JSON in about 60 measurement paths. Ratings, HP and shield values used to be doubles and are now int64 on the wire (some old HP values were fractional). coords and the *_level_percentages used to be integral ints and are now float32 (-515.0). The consumer I checked tolerates this: re-running 8 real converted journals with 7,641 measurement values flipped between int and float gave identical persisted results. What it does depend on is the integer markers in battle_log (-98 is recognized, -98.0 is not; floating them took an armada from 299 decoded attacks to zero) and integer ids. Your test already pins exactly that, so thank you. It might be worth a sentence for sync-site maintainers that measurement fields can change between int and float while ids and battle_log markers stay integers.

Tests

  • Windows, Zig 0.16.0, MSVC 14.51.36231 (STL update 202604): zig build test-sync -Dtarget=x86_64-windows-msvc passes, including protobuf journals. In case it helps narrow down the build-win STL1000 failure, this toolset works locally.

In game (1.1.10 release layout)

  • I ported it to the 1.1.10 layout the same way: journal.{h,cc} verbatim, the 10 messages applied unchanged (the 1.1.10 Digit.PrimeServer.Models.proto is identical to dev's), and the same JSON/protobuf branch in the journal fetch. In one session, which also re-sent a backlog after I cleared the sent-battles file, a private consumer accepted 48 battles: 47 hostile and one armada (type 8). The armada decoded completely (20 rounds, 299 attacks, 124 ability records, no unknown tags), so that is live type 8 coverage. It had a single player ship, so multi-participant armada mapping is still untested. The mod log had no conversion errors.
  • Two notes for anyone backporting to 1.1.10:
    • The participant collector there indexes a const json with operator[] (deployed_fleets, uid). Converted journals can leave keys out, so it should use find() the way Participants on dev does.
    • Battle ids are recorded as sent when they are queued, before the journal fetch, so battles since 2026-11-19 are not retried after upgrading. Deleting patch_battlelogs_sent.json once, with the game closed, backfills whatever the game still lists as recent. I haven't checked how dev's record store behaves here.

The validation and this write-up were done with help from Claude Code.

@blevouschurverantunt

Copy link
Copy Markdown
Contributor Author

Thanks for the independent check, and for the live armada.

  • Widths: the message descriptors embedded in the client metadata declare int64 for both slot_index and the system_buffs values, so I kept them. Decoding is the same either way, and I can narrow them if a maintainer prefers the C# types.
  • state_updated_at: added to the omitted-keys list in the description.
  • Number kinds: added a sentence for sync-site maintainers.

@netniV
netniV added this pull request to the merge queue Nov 22, 2026
Merged via the queue into STFC-Mod:dev with commit 8d0d31b Nov 22, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants