Repository navigation
fix(models): count tool calls and results when sizing a context cache - #8234
Open
karthik-0306 wants to merge 2 commits into
Open
karthik-0306 wants to merge 2 commits into
karthik-0306 wants to merge 2 commits into
Conversation
The size gate added in google#6137 estimates the cacheable prefix's share of the previous prompt from character counts, but _estimate_request_tokens only counted part.text. Function calls and responses counted as zero, so in a tool loop a large trailing function response let a tiny prefix through to caches.create (400 INVALID_ARGUMENT), and a large one inside the prefix kept caching from ever turning on. Count their names and JSON payloads, and code parts, in the estimate. Fixes google#8225
Media parts counted as zero in the prefix estimate, so a large trailing image or document, or one a tool returned in FunctionResponse.parts, could still let a tiny prefix through the size gate. Count each media item as one small image (258 tokens), and cover code and media parts in the regression tests. Fixes google#8225
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
The size gate added in #6137 estimates the cacheable prefix's share of the previous prompt from character counts, but
_estimate_request_tokensonly countedpart.text. Function calls and function responses counted as zero, which breaks the estimate in a tool loop in both directions:caches.createfailed with400 INVALID_ARGUMENT.Solution:
Count each part's payload in the estimate:
args/responseexecutable_codeandcode_execution_resultoutputThe model tokenizes these much like text. Media (
inline_data,file_data) is still not counted, because there is no character-based proxy for its token cost. When a prompt has no countable parts at all, the existing fallback to the real token count still applies. The extra JSON encoding only runs on the cache-creation path, not on every request.Testing Plan
Unit Tests:
Two tests added to
tests/unittests/agents/test_gemini_context_cache_manager.py, one for each direction. Both fail onmainand pass with this change:test_large_trailing_tool_result_does_not_pass_cache_size_gatetest_large_cached_tool_result_passes_cache_size_gatepytest results:
test_gemini_context_cache_manager.py: 54 passed.tests/unittests/{models,agents,flows}: 2587 passed. The remaining 1 failure and 13 collection errors are identical on unmodifiedmainin my environment (optional extras not installed) and don't touch these files.ruff,isort,pyink,codespellandmypyare clean on the changed files.Manual End-to-End (E2E) Tests:
Ran the repro script from #8225 (a function call followed by a large function response, with the context cache config on
gemini-2.5-flash):estimated prefix tokens: 30050,caches.create calls: 1(rejected by the API).caches.createis not called.Checklist
Additional context
The change is limited to
_estimate_request_tokensand a small helper. The cache fingerprinting, the TTL handling and the min-token thresholds are unchanged.