Repository navigation
from_diff: avoid the generic deepcopy and the sorted copy for move keys - #274
Merged
stefankoegl merged 2 commits intoNov 19, 2026
Merged
Conversation
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Equal non-native containers can turn moves into remove/add operations that fail serialization, while uncopied dictionary keys retain shared mutable state with the input and break key–value aliasing.
2 open issues
What changed in this PR
This PR reduces diff-creation overhead in JsonPatch.from_diff by optimizing operation copying and move-key serialization.
Changes:
- Replaces generic deep copying with a memoized JSON-specific copier that falls back to
copy.deepcopyfor other types. - Uses a cached, key-sorting JSON encoder for default move detection.
- Retains the existing serialization path for custom
dumpscallables.
| File | Description |
|---|---|
jsonpatch.py |
Adds the specialized copier and cached encoder, and updates both diff entry points. |
Review effort: Balanced
Comment on lines
+1214
to
+1215
| except (TypeError, ValueError, RecursionError): | ||
| return id(value) |
| # that self-referential documents terminate like copy.deepcopy | ||
| copied = memo[value_id] = {} | ||
| for key, item in value.items(): | ||
| copied[key] = _json_copy(item, memo) |
The cached sort_keys encoder cannot serialize UserDict and UserList values, so equal but independently allocated values got different move keys, a move became remove plus add, and to_string() then failed to serialize the added value. Fall back to the generic _sorted_members path for values the encoder rejects; it normalizes those containers the same way the code did before the encoder was cached. _json_copy also reused the dictionary keys instead of copying them. Copy them through the same memo as the values, like copy.deepcopy does, so that a key that is also a value stays one object and a str subclass used as a key is not aliased with the input document. Adds a regression test for each case.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

JsonPatch.from_diffdeep-copies the whole operation list and, for movedetection, builds a sorted deep copy of every added or removed value before
serializing it. On large documents these two defensive copies dominate diff
creation: on a real 891 KB -> 3.03 MB Botocore service model pair
(
ec2/2016-09-15->ec2/2016-11-15, 5770 generated operations),copy.deepcopyalone is about 30% of the runtime and the sorted-copy stepabout 20%.
Change (one area, 50 changed lines in
jsonpatch.pyplus two regressiontests, no API or output change):
copy.deepcopy. It handlesdict/list/scalars directly, keeps a memo soshared and self-referential values behave exactly like
deepcopy, and fallsback to
copy.deepcopyfor any other type, so the documented "values in thepatch are copied" guarantee is preserved.
sort_keys=Trueinstead of firstmaterialising
_sorted_members(value)and then serializing it, and cachejson.JSONEncoder(sort_keys=True)for the defaultjson.dumps. Customdumps=callables keep the old code path.Verification (all run locally):
348 passed, 448 subtests passed(346 before the reviewfollow-up; the two new tests fail on the previous commit).
flake8 jsonpatch.py tests.pyclean,coverage run tests.pyandcoverage run property_tests.pypass.seeded random JSON + structural mutations of real Botocore documents + edge
cases such as empty documents,
truevs1,1vs1.0,-0.0,non-ASCII keys, NUL bytes and 4 KiB strings) produce byte-identical
patch.to_string()output; 0 mismatches, and all 1200 patches re-apply todst. The upstream suite's self-referential-value test also passes.document pairs, and 1.34-1.40x median over the three size scales across
four full bench runs. Peak RSS
1.008x (
/usr/bin/time -v).Follow-up after the review in this PR:
_move_keynow falls back to the_sorted_memberspath when the cachedencoder rejects a value.
UserDictandUserListare normalized there, soequal but independently allocated values keep equal move keys. Before, they
fell back to
id(), a move became remove plus add, andto_string()failedon the added value.
_json_copycopies dictionary keys through the same memo as the values,matching
copy.deepcopy: a key that is also a value stays one object, and astrsubclass used as a key is not aliased with the input document.UserDict,UserList, tuple, shared-reference andstrsubclass keys)produces byte-identical
to_string()output and the same error classes asthe unpatched upstream module in all 2500 cases.
One benign difference from the unpatched module remains for non-JSON
tuple values: the
sort_keysencoder also sorts object members inside atuple, so
[({'b': 1, 'a': 2},), 'X']->['X', ({'a': 2, 'b': 1},)]isdetected as a move where upstream emitted remove plus add. Both patches
serialize and apply correctly; tuples are not JSON.
The one honest negative: for two nearly identical 3 MB models (274 operations)
the speedup is 1.02x, because the saved work is proportional to the number of
added/removed subtrees rather than to input size. This is a diff-creation
fast path, not a general parsing speedup.
This patch was prepared with AI assistance (DeepSeek Harness) and reviewed against the upstream test suite before submission.