Skip to content

fix(models): count tool calls and results when sizing a context cache - #8234

Open
karthik-0306 wants to merge 2 commits into
google:mainfrom
karthik-0306:fix/8225-cache-estimate-non-text-parts
Open

karthik-0306 wants to merge 2 commits into
google:mainfrom
karthik-0306:fix/8225-cache-estimate-non-text-parts

Conversation

@karthik-0306

Copy link
Copy Markdown

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
The size gate added in #6137 estimates the cacheable prefix's share of the previous prompt from character counts, but _estimate_request_tokens only counted part.text. Function calls and function responses counted as zero, which breaks the estimate in a tool loop in both directions:

  • A large trailing function response (outside the prefix) made a tiny prefix look like the whole prompt. The gate let it through and caches.create failed with 400 INVALID_ARGUMENT.
  • A large function response inside the prefix made the prefix look tiny, so caching never turned on.

Solution:
Count each part's payload in the estimate:

  • function call and function response names plus their JSON-encoded args / response
  • executable_code and code_execution_result output

The model tokenizes these much like text. Media (inline_data, file_data) is still not counted, because there is no character-based proxy for its token cost. When a prompt has no countable parts at all, the existing fallback to the real token count still applies. The extra JSON encoding only runs on the cache-creation path, not on every request.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

Two tests added to tests/unittests/agents/test_gemini_context_cache_manager.py, one for each direction. Both fail on main and pass with this change:

  • test_large_trailing_tool_result_does_not_pass_cache_size_gate
  • test_large_cached_tool_result_passes_cache_size_gate

pytest results:

  • test_gemini_context_cache_manager.py: 54 passed.
  • tests/unittests/{models,agents,flows}: 2587 passed. The remaining 1 failure and 13 collection errors are identical on unmodified main in my environment (optional extras not installed) and don't touch these files.
  • ruff, isort, pyink, codespell and mypy are clean on the changed files.

Manual End-to-End (E2E) Tests:

Ran the repro script from #8225 (a function call followed by a large function response, with the context cache config on gemini-2.5-flash):

  • Before: estimated prefix tokens: 30050, caches.create calls: 1 (rejected by the API).
  • After: the prefix is estimated at about 15 tokens, below the 2,048 / 4,096 minimum, and caches.create is not called.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

The change is limited to _estimate_request_tokens and a small helper. The cache fingerprinting, the TTL handling and the min-token thresholds are unchanged.

The size gate added in google#6137 estimates the cacheable prefix's share of the
previous prompt from character counts, but _estimate_request_tokens only
counted part.text. Function calls and responses counted as zero, so in a tool
loop a large trailing function response let a tiny prefix through to
caches.create (400 INVALID_ARGUMENT), and a large one inside the prefix kept
caching from ever turning on. Count their names and JSON payloads, and code
parts, in the estimate.

Fixes google#8225
Media parts counted as zero in the prefix estimate, so a large trailing image
or document, or one a tool returned in FunctionResponse.parts, could still let
a tiny prefix through the size gate. Count each media item as one small image
(258 tokens), and cover code and media parts in the regression tests.

Fixes google#8225
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Context cache size gate ignores function call/response parts, so caches.create still fails with 400 INVALID_ARGUMENT in tool loops

2 participants