Repository navigation
feat(agents): add a Gator review team and review replay harness - #5779
johntmyers wants to merge 2 commits into
Conversation
Add bin/review-team, which runs the generalist reviewer plus focused members chosen by review mode and changed code, merges their findings, and checks that every member finding is accounted for. Networking and contracts members run by default; correctness and security are opt-in. Add a replay harness that reruns a reviewer on pinned past PRs with known later fixes and grades the findings, to measure reviewer changes. Gator's review step is unchanged until the team is wired in. Closes #5778 Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
6bdfcd7 to
81127e6
Compare
…reviews Add networking checklist rules: control-plane changes such as policy reloads, approvals, and credential rotations disrupt only the connections they affect, and a targeted check that replaces a conservative one reads consistent, fully published state with versions that change on every transition. The contracts member now compares the pull request description's claims with the behavior the diff changes. Add replay cases for #2373, which closed every relay on each policy reload, and #5361, whose credential revalidation raced publication. Signed-off-by: John Myers <9696606+johntmyers@users.noreply.github.com>
Karthik-Chowdary
left a comment
There was a problem hiding this comment.
Bound reviewer shutdown after the replay timeout
At b46f31f, the watchdog in .agents/agents/gator/evals/run-case:91–105 sends SIGTERM once and exits, while the parent still waits without a deadline. A reviewer that ignores SIGTERM or stalls during shutdown therefore keeps its case running and prevents run-replay from reaching scoring.
I reproduced this with the unmodified run-case and its real task-builder helpers, using an offline reviewer command and a 1-second timeout: the cooperative control finished in about 2.6 seconds; the TERM-resistant reviewer was still alive after 6 seconds, with the timeout message already logged but no result.json. Only external SIGKILL let the case finish. This tests the documented custom-reviewer path, not normal Codex signal handling.
Could timeout shutdown escalate after a bounded grace period and cover a TERM-resistant reviewer in the offline replay test? The unchanged review_team_test.sh and replay_test.sh pass with jq 1.8.1 and Ruby 3.0.2.
Summary
Add a Gator review team (
bin/review-team) and a replay harness that measures reviewer recall on past PRs with known later fixes. Gator's review step is unchanged; a follow-up wires the team into gator-gate.Related Issue
Closes #5778
Changes
bin/review-teamruns the generalist reviewer plus focused members chosen by review mode and changed code, in parallel through the harness subagent adapter. A merge subagent combines findings, and a deterministic check requires every member finding to be merged or dropped with a reason. A member or merge failure (after one retry) fails the run without output.review-networkingruns when changed paths or added code touch networking, and must give a verdict on each of the 35 rules innetworking-checklist.md.review-contractsruns in initial and follow-up reviews.review-correctnessandreview-securityare opt-in throughREVIEW_TEAM_MEMBERS.highreasoning.OPENSHELL_SUBAGENT_REASONING, so the team does not depend on a specific harness's reasoning setting.evals/holds 27 pinned cases (Gator review misses and networking escapes).run-replaychecks out each PR head without later history, builds the reviewer task from the gator-gate skill, runs the reviewer or the team, and grades findings with a judge model.scorecompares runs.agent.yamladds the member prompts (prompts/review-*.md), member rules, checklist, and team config to the payload.payload_versionis unchanged because nothing uses them yet.On the 19 small and medium replay cases (21 known bugs), each graded by the same judge:
Each configuration ran once, so differences of one or two catches are within run-to-run noise.
Networking checklist rules 34 and 35 and the contracts member's claims-against-the-diff check come from #2373, which closed every relay on each policy reload, and #5361, whose credential revalidation raced policy publication. Three runs of the default team on each, before and after these rules:
Testing
review_team_test.sh,replay_test.sh, Codexexec_test.sh,run_test.sh,launch_test.sh, andimage_test.shpassreview_team_test.sh,replay_test.sh)Checklist