Skip to content

from_diff: avoid the generic deepcopy and the sorted copy for move keys - #274

Merged
stefankoegl merged 2 commits into
stefankoegl:masterfrom
linluozi:from-diff-copy-fastpath
Nov 19, 2026
Merged

stefankoegl merged 2 commits into
stefankoegl:masterfrom
linluozi:from-diff-copy-fastpath

Conversation

@linluozi

@linluozi linluozi commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

JsonPatch.from_diff deep-copies the whole operation list and, for move
detection, builds a sorted deep copy of every added or removed value before
serializing it. On large documents these two defensive copies dominate diff
creation: on a real 891 KB -> 3.03 MB Botocore service model pair
(ec2/2016-09-15 -> ec2/2016-11-15, 5770 generated operations),
copy.deepcopy alone is about 30% of the runtime and the sorted-copy step
about 20%.

Change (one area, 50 changed lines in jsonpatch.py plus two regression
tests, no API or output change):

  1. copy the emitted operation list with a small JSON-specific copier instead of
    copy.deepcopy. It handles dict/list/scalars directly, keeps a memo so
    shared and self-referential values behave exactly like deepcopy, and falls
    back to copy.deepcopy for any other type, so the documented "values in the
    patch are copied" guarantee is preserved.
  2. serialize move-detection keys with sort_keys=True instead of first
    materialising _sorted_members(value) and then serializing it, and cache
    json.JSONEncoder(sort_keys=True) for the default json.dumps. Custom
    dumps= callables keep the old code path.

Verification (all run locally):

  • upstream suite: 348 passed, 448 subtests passed (346 before the review
    follow-up; the two new tests fail on the previous commit).
  • flake8 jsonpatch.py tests.py clean, coverage run tests.py and
    coverage run property_tests.py pass.
  • differential: 1200 independent (src, dst) pairs (de-duplicated by sha256;
    seeded random JSON + structural mutations of real Botocore documents + edge
    cases such as empty documents, true vs 1, 1 vs 1.0, -0.0,
    non-ASCII keys, NUL bytes and 4 KiB strings) produce byte-identical
    patch.to_string() output; 0 mismatches, and all 1200 patches re-apply to
    dst. The upstream suite's self-referential-value test also passes.
  • benchmark on 10 real Botocore files (8.43 MB): median speedup 1.33x over five
    document pairs, and 1.34-1.40x median over the three size scales across
    four full bench runs. Peak RSS
    1.008x (/usr/bin/time -v).

Follow-up after the review in this PR:

  • _move_key now falls back to the _sorted_members path when the cached
    encoder rejects a value. UserDict and UserList are normalized there, so
    equal but independently allocated values keep equal move keys. Before, they
    fell back to id(), a move became remove plus add, and to_string() failed
    on the added value.
  • _json_copy copies dictionary keys through the same memo as the values,
    matching copy.deepcopy: a key that is also a value stays one object, and a
    str subclass used as a key is not aliased with the input document.
  • A 2500 case differential (seeded; plain and mutated JSON documents with
    UserDict, UserList, tuple, shared-reference and str subclass keys)
    produces byte-identical to_string() output and the same error classes as
    the unpatched upstream module in all 2500 cases.

One benign difference from the unpatched module remains for non-JSON
tuple values: the sort_keys encoder also sorts object members inside a
tuple, so [({'b': 1, 'a': 2},), 'X'] -> ['X', ({'a': 2, 'b': 1},)] is
detected as a move where upstream emitted remove plus add. Both patches
serialize and apply correctly; tuples are not JSON.

The one honest negative: for two nearly identical 3 MB models (274 operations)
the speedup is 1.02x, because the saved work is proportional to the number of
added/removed subtrees rather than to input size. This is a diff-creation
fast path, not a general parsing speedup.


This patch was prepared with AI assistance (DeepSeek Harness) and reviewed against the upstream test suite before submission.

@coveralls

coveralls commented Nov 13, 2026 •

Copy link
Copy Markdown

Coverage Status

coverage: 100.0%. remained the same — linluozi:from-diff-copy-fastpath into stefankoegl:master

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Equal non-native containers can turn moves into remove/add operations that fail serialization, while uncopied dictionary keys retain shared mutable state with the input and break key–value aliasing.

2 open issues
What changed in this PR

This PR reduces diff-creation overhead in JsonPatch.from_diff by optimizing operation copying and move-key serialization.

Changes:

  • Replaces generic deep copying with a memoized JSON-specific copier that falls back to copy.deepcopy for other types.
  • Uses a cached, key-sorting JSON encoder for default move detection.
  • Retains the existing serialization path for custom dumps callables.
File Description
jsonpatch.py Adds the specialized copier and cached encoder, and updates both diff entry points.

Review effort: Balanced

Comment thread jsonpatch.py Outdated
Comment on lines +1214 to +1215
except (TypeError, ValueError, RecursionError):
return id(value)
Comment thread jsonpatch.py Outdated
# that self-referential documents terminate like copy.deepcopy
copied = memo[value_id] = {}
for key, item in value.items():
copied[key] = _json_copy(item, memo)
The cached sort_keys encoder cannot serialize UserDict and UserList
values, so equal but independently allocated values got different move
keys, a move became remove plus add, and to_string() then failed to
serialize the added value. Fall back to the generic _sorted_members
path for values the encoder rejects; it normalizes those containers the
same way the code did before the encoder was cached.

_json_copy also reused the dictionary keys instead of copying them.
Copy them through the same memo as the values, like copy.deepcopy does,
so that a key that is also a value stays one object and a str subclass
used as a key is not aliased with the input document.

Adds a regression test for each case.
@stefankoegl
stefankoegl merged commit 84a64f5 into stefankoegl:master Nov 19, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants