Skip to content

Heartbeats only inside ctx.working; a shutdown is the default - #322

Merged
igalshilman merged 6 commits into
mainfrom
claude/opt-in-heartbeats
Nov 19, 2026
Merged

igalshilman merged 6 commits into
mainfrom
claude/opt-in-heartbeats

Conversation

@igalshilman

@igalshilman igalshilman commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

A session object's attempt heartbeated on its own while nothing of Restate's was outstanding, or while an annotation was open. That had three problems:

  • a handler stuck on a promise of its own was kept alive forever;
  • the OpenAI adapter's annotation around every tool call kept the attempt alive through a tool waiting for a signal, for as long as the wait lasted;
  • it decided the default for the handler.

Heartbeats are now opt-in. Suspension is the default. This is the same change the TypeScript SDK made.

sessions

  • ctx.working(), an async context manager.
    • While any scope is open, the attempt heartbeats, whatever it waits for.
    • Outside a scope, Restate shuts the attempt down at its inactivity timeout. It suspends on its calls, or starts again at once if it waits for none.
    • It must be entered on the attempt's own event loop.
  • The driver counts open scopes. A scope's first heartbeat comes once the attempt has been silent for an interval (at once, if it already has), so the silence never exceeds one interval, and the database's per-call scope costs no message.
  • The database's own calls run in a scope. That covers open, statements and commits.
  • Annotations are traces only. They keep nothing alive.
  • API: the SessionContext protocol gains working(), so a test's own fake context needs it too (contextlib.nullcontext() will do).

agents

  • tool(keepalive=True) runs a tool of your own in ctx.working(). By default a tool is not kept alive, since it may wait on Restate.
  • Every adapter keeps its model requests alive, and the summarizer's, through the framework's own hooks, with no library patched:
Adapter Model requests Built-in local tools kept alive
OpenAI from the end of on_llm_start to the top of on_llm_end the SDK's own tools (shell, local shell, apply_patch, computer)
Pydantic AI a WrapperModel around the run's model, streamed or not; a compaction's too the harness's file system (by kind) and shell (by its exported tool names)
ADK the plugin, from after the steering to the whole response or an error the shell, the skills' scripts, the environment's tools
LangGraph the built-in agent's request; a user graph's via an inherited callback handler, async or sync (invoke in a sync node) none (see below)
  • Approval, deferred and interrupt waits always stay outside.
  • Scopes a run leaves open are closed when the run ends.

Known gaps

  • OpenAI: an agent used as a tool of another (Agent.as_tool). The SDK gives its nested run no hooks.
  • ADK: an agent's code executor, which runs as a processor of the flow, not as a tool.
  • LangGraph: the shell of LangChain's ShellToolMiddleware. It is a plain tool with nothing that tells it apart.

Testing

  • The whole suite passes: 816 tests, including the server tests. Ruff, pyright and mypy are clean.
  • sessions:
    • the driver's heartbeat tests follow the new rule;
    • the server tests check that work in ctx.working() keeps its attempt, and that work outside it is cut off and restarted.
  • agents:
    • each adapter checks the open scopes with the test Host: model request, plain tool, keepalive tool, built-in local tool, approval or interrupt wait, after the turn, and failed or stopped runs;
    • a LangGraph sync node's request, with a working() that starts a task as the driver's does;
    • the long-work and dead-attempt server tests run their scripted model in ctx.working().

An attempt heartbeated on its own while nothing of Restate's was outstanding,
or while an annotation was open. That kept a handler stuck on a promise of its
own alive forever, and an annotation around a tool that waits for a signal
held its process for the whole wait.

- `ctx.working()`: an async context manager. While any scope is open the
  attempt heartbeats, whatever it waits for; outside, Restate shuts it down at
  its inactivity timeout and it suspends on its calls (or starts again at once,
  waiting for none).
- The driver counts the scopes. A scope's first heartbeat comes at once only if
  the attempt wrote nothing for an interval, so the database's scope per call
  costs no message.
- The database's own calls (open, statements, commits) run in a scope.
- Annotations are traces only: they keep nothing alive.
- `tool(keepalive=True)` runs a tool of your own in `ctx.working()`; by
  default a tool is not kept alive.
- Tests: the driver's and the server's heartbeat tests follow the new rule;
  the scripted frameworks of the long-work and dead-attempt tests run their
  model in `ctx.working()`; the test `Host` counts open scopes.
…tools run kept alive

- OpenAI: each model request is kept alive from the end of `on_llm_start` to
  the top of `on_llm_end`, so the steering and commits around it are not. The
  SDK's own tools (shell, local shell, apply_patch, computer) are kept alive
  per call; a function tool is the user's, and is not. The summarizer's
  request too. Whatever a run leaves open is closed as it ends.
- Pydantic AI: the model the run gets is wrapped, so exactly each request
  (streamed or not) is kept alive; a compaction's too. The harness's file
  system tools (by their kind) and its shell (by the names it gives its tools)
  are kept alive per call; `code_mode` is not, since it runs the user's tools.
  Deferred and approval waits stay outside.

Known gap: the model requests of an OpenAI agent used as a tool of another
(`Agent.as_tool`) are not kept alive: the SDK gives its nested run no hooks.
…s built-in local tools

- ADK: the plugin keeps each model request alive from the end of its
  `before_model_callback` (after the steering) to the whole response, an
  error, or the event that ends the step if a callback answered instead. The
  shell, the skills' scripts and the environment's tools (known by their
  package, as ADK keeps their classes private) are kept alive per call. The
  summarizer's request too. A run closes what is left as it ends. Not
  covered: an agent's code executor, which runs as a processor of the flow.
- LangGraph: the built-in agent's request and its summarizer's run in
  `ctx.working()`. A graph of your own gets a callback handler, inherited by
  its nodes' runs, that keeps each chat model request alive from its start
  (awaited before it is sent) to its end or error, and releases what is left
  before any wait for an interrupt.

Known gap: the shell of LangChain's `ShellToolMiddleware` is a plain tool
with nothing that tells it apart, so it is not kept alive.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

In a user LangGraph graph, a sync node that calls model.invoke now hangs forever before its request goes out. I reproduced it.

Approach: sound. Making keepalive opt-in with a scope counter in the driver fixes the cause rather than tuning the old heuristic. Wiring each adapter through its own framework hooks (no patching) is the right place for it. The one exception is LangChain's sync callback bridge, which the _KeepAlive handler doesn't account for.

Reviewed changes: I read the full diff of all 4 commits: the sessions driver and database, the four agent adapters, the docs, and the tests.

  • ctx.working() and the driver's scope count: heartbeats now run only while a scope is open. The first beat comes at once only if the attempt has been silent for a full interval.
  • Annotations are traces only: open_annotation/close_annotation no longer affect heartbeats, and _store_waits/_awaits_restate are gone.
  • Database scopes: sql_open, each store request and commit run in the host's database_working.
  • tool(keepalive=True): the tool body runs in ctx.working(). Plain tools are no longer kept alive.
  • OpenAI adapter: model requests are scoped from on_llm_start to on_llm_end, and the SDK's non-FunctionTool tools are scoped per call. ended() is now async and closes leftover scopes.
  • Pydantic AI adapter: a _Alive WrapperModel wraps the run model and the compaction model. The harness's file-system kinds and shell names are kept alive.
  • ADK adapter: plugin hooks scope each request per (invocation, branch, agent), plus the built-in local tools. release_all runs in the run's finally.
  • LangGraph adapter: the built-in agent and the summarizer are wrapped directly. User graphs get an inherited _KeepAlive callback handler.
  • Tests: each adapter checks open scopes through the test Host. The driver and server tests follow the new rule.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread python/restate/sessions/_driver.py Resolved
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Test Results

  8 files  ±0    8 suites  ±0   3m 52s ⏱️ +17s
 71 tests ±0   71 ✅ ±0  0 💤 ±0  0 ❌ ±0 
322 runs  ±0  322 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 1c01bee. ± Comparison against base commit fc7eb19.

♻️ This comment has been updated with latest results.

…rtbeat comes within an interval

- LangChain runs the callbacks of a sync request (`invoke`, in a sync node) on
  a loop of its own, in a thread, and waits there for every task they start.
  The handler entered `ctx.working()` there, whose heartbeat task never ends:
  the request never went out. It now opens and closes its scopes on the
  attempt's loop, handed over without waiting; one that arrives after its
  stream ended is dropped. A test with a sync node, whose `working()` starts a
  task as the driver's does, hung before.
- `ctx.working()` refuses to be entered on another loop than the attempt's.
- A scope's first heartbeat waited a whole interval after any recent write, so
  the silence could reach almost two intervals; it now comes once the silence
  reaches one.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Both earlier findings are fixed. One minor wording suggestion is inline.

Reviewed changes: Since e5f94ac, one commit (23cb6bb) fixes the sync LangGraph request hang and shortens the first heartbeat's delay.

  • _KeepAlive loop handoff: the handler captures the attempt's loop. It runs _open/_close inline on that loop, and from any other loop it hands them over with run_coroutine_threadsafe without waiting. Because neither suspends before it touches the driver's scope, they stay in order. A _stream counter drops an open that arrives after release_all.
  • Driver guard: working() raises if it is entered off the attempt's loop. Every in-repo caller (the adapters, tool(keepalive=True), the database boundary) is async on that loop.
  • First heartbeat: the first sleep is now only the time left in the interval, so the silence never exceeds one interval and the turns.md statement holds.
  • Test: a sync node whose test working() spawns a task the way the driver does. It passes locally along with the driver's heartbeat tests.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread python/restate/sessions/_driver.py Resolved

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. Every earlier finding is addressed.

Reviewed changes: One commit since 23cb6bb.

  • Off-loop error wording: the RuntimeError that ctx.working() raises off the attempt's event loop now states the requirement ("must be entered on the attempt's own event loop").

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@igalshilman
igalshilman merged commit 570960b into main Nov 19, 2026
12 checks passed
@igalshilman
igalshilman deleted the claude/opt-in-heartbeats branch November 19, 2026 16:19
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants