Skip to content

Pace archive lookups, back off on refusals, and check links in threads - #540

Open
JesperDramsch wants to merge 3 commits into
mainfrom
ai-agent/gallant-dirac-yvp0kp
Open

JesperDramsch wants to merge 3 commits into
mainfrom
ai-agent/gallant-dirac-yvp0kp

Conversation

@JesperDramsch

@JesperDramsch JesperDramsch commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Why

The first Archive Resolve run (15 November, run 43705009939) found 270 yearless URLs on superseded editions and 127 broken ones. It replaced 65 of them, and those reached main through #535. Then the archive started refusing connections. Of the 301 links it left in place, 275 were failed lookups and only 26 had no capture. The refusals came in waves: about 20 successes, then runs of 20 or more refusals, each followed by more successes. The run kept sending a request every second through all of them. It also ended red: the list of kept links would have made the rolling PR body 165,880 characters, and GitHub's limit is 65,536. It took 38 minutes of Actions time.

What changes

Fewer CDX requests (utils/wayback.py).

  • host_captures() lists the captures of several pages on one host in a single request, url=<host>/*. A filter on the archive's side (filter=original:<regex>) keeps the answer to just those pages. Checked by hand against pycon.ru: the filtered answer holds only /, /cfp and /partneram, including the www. variants, while the unfiltered one lists speaker pages, robots.txt and a sitemap.
  • No collapse. CDX collapses adjacent lines, so in a whole-host answer one page's capture could hide another page's capture from the same day.
  • page_key() matches captures to the data the way CDX files them: without www., scheme, trailing slash or #fragment.
  • Hosts that carry many unrelated sites (SHARED_HOSTS: Google Forms and Drive, pydata.org, pretalx, Sessionize, GitHub, Linux Foundation and others) stay one page per request.
  • On today's backlog this means 146 requests for 185 URLs. Most hosts in it have only one candidate, so grouping saves less than the pacing does.

Pacing, backoff and a time limit (utils/archive_resolve.py).

  • Requests go out one at a time, 1 s apart, with 5 s instead after every 10th (--spacing, --delay).
  • A refused connection or HTTP 429/509 raises WaybackBlockedError, the CDX counterpart of what Stop the archive save run when the Wayback Machine turns it away #539 does for saves. The same request is retried after 10, 20 and 40 s. If it's still refused, the run stops asking, and the next run finds the remaining links again.
  • Each new refusal restarts the backoff from 10 s. All backoff waits in one run are capped at 180 s.
  • The lookups stop after 600 s in all (TIME_BUDGET) and leave the rest for the next run, the same as after a refusal. A slow archive that still answers can't run the step into its timeout and lose the replacements already found.
  • Other failures, such as a 503 for one host, only mark those URLs and the run carries on.

Only one Wayback job at a time. archive-new-urls.yml and archive-resolve.yml share a job-level wayback concurrency group, so they never send the archive requests at once from two runners. A pending job can be replaced by a newer one. For the save job that loses nothing, because it starts from its last successful run. A replaced resolve run waits until the next week.

NEEDS:archive only lists what needs a person. It holds the links with no capture from their edition's time, 50 at most, with a count of the rest. Failed and deferred lookups aren't listed, because the next run retries them. This keeps the body under GitHub's limit.

Fewer Actions minutes in the link check.

  • URLs are requested in 16 threads inside the single runner, at most 2 at a time per site, with a 5 s timeout instead of 10 s. There's no matrix and no extra jobs.
  • URLs are handed to the threads round-robin across hosts, so long runs on one big host don't hold most of the workers on its two slots.
  • The threads only reach the conference sites, never the Wayback Machine.
  • The job is capped at 30 minutes instead of 180.
  • PYTHONUNBUFFERED=1 makes the log show where the time goes.
  • The job summary shows the run's counts: replaced, no capture, failed, left for the next run, Mastodon moves, redirects.

Tests

  • New: TestLookupPlan, TestResolveByHost, TestTurnedAway, TestTimeBudget, TestBatches, TestNeedsList, TestRequestAll, TestHostCaptures, TestPageKey, and the WaybackBlockedError cases for CDX.
    • TestTurnedAway covers backing off 10/20/40 s then stopping, carrying on once a refusal lifts, restarting the backoff for each refusal, the 180 s cap, and other failures not stopping the run.
    • TestBatches checks 1 s within a batch and 5 s after every 10th request.
    • TestRequestAll checks every URL gets answered, at most PER_HOST requests run per site at once, and hosts are interleaved.
  • pytest tests/ (REVIEW.md's uv line): 2664 passed, 4 skipped, 3 xfailed, 2 xpassed.
  • pre-commit (isort, black, ruff, docsig, pyupgrade) is clean on the changed files.

Not run: the workflows themselves. They run next on their crons, or by hand with workflow_dispatch once this is merged.

🤖 Generated with Claude Code

https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt

The first Archive Resolve run (15 Nov) replaced 65 links, then the
archive started refusing connections: 275 of the 301 links it kept were
failed lookups, and the run kept asking one a second through all of
them. It also ended red because the list of kept links made the rolling
PR body 165,880 characters, over GitHub's 65,536.

- wayback: a refused connection or HTTP 429/509 on a CDX request raises
  WaybackBlockedError. host_captures() lists several pages of one host
  in a single request, with a filter on the archive's side so the
  answer holds only those pages; page_key() matches captures to the
  data as CDX does (no www., scheme, trailing slash or #fragment).
- archive_resolve: pages of a host with several candidates share one
  request; hosts that carry many sites (SHARED_HOSTS) stay one page
  per request. At the first refusal the run stops asking, without
  waiting, and leaves the rest for the next run, which finds them again.
  NEEDS:archive lists only links with no capture from their edition's
  time, at most 50; failed and deferred lookups are retried, not listed.
- The link check requests URLs in 16 threads of the one runner, at most
  two at a time per site, with a 5 s timeout instead of 10 s.
- The workflow prints as it runs, writes the run's counts to the job
  summary, and caps the job at 30 minutes instead of 180.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt

Copy link
Copy Markdown
Owner Author

Routine review

1. Request

The PR body is the request, and no issue is linked. It asks for four changes after the 15 November Archive Resolve run:

  • In utils/wayback.py: batch CDX lookups per host with host_captures() and page_key(), and add SHARED_HOSTS as the exceptions.
  • In utils/archive_resolve.py: stop at the first archive refusal (WaybackBlockedError on a refused connection or HTTP 429/509) and defer the rest without sleeping.
  • Cap NEEDS:archive at 50 lines of no-capture links only.
  • Speed up the link check: 16 threads, 2 per host, a 5 s probe timeout, a 30/20 min job/step timeout, PYTHONUNBUFFERED and a counts summary in .github/workflows/archive-resolve.yml.

2. Checked against

  • Date: Thu Nov 19 10:13:47 UTC 2026
  • Wayback CDX server README, https://github.com/internetarchive/wayback/blob/master/wayback-cdx-server/README.md: "Collapsing is done on adjacent cdx lines where all captures after the first one that are duplicate are filtered out"; url=archive.org/* is "equivalent to url=archive.org/&matchType=prefix". The README doesn't say whether filters run before collapse, or whether www. falls under a prefix query.
  • unverified: the PR body's hand check against pycon.ru ("the filtered answer holds only /, /cfp and /partneram, including the www. variants"). web.archive.org is blocked from this sandbox (CONNECT tunnel failed, response 403).
  • unverified: "146 requests for 185 URLs" and the GitHub PR body limit of 65,536 characters.

3. Needs your decision

None. Everything the diff does is named in the PR body.

4. Findings

Important: None.

Nits:

  • utils/wayback.py (host_captures, "collapse": "timestamp:8"), judgement call. Per the CDX README, collapse acts on adjacent lines. In a host/* answer, the last capture of one page and the first capture of the next page can sit next to each other. When they fall on the same day, the second page's capture is dropped. If that was the page's only capture before end+14 days, the page is reported no-capture when a capture exists. The cost is a kept link listed for a person, not wrong data. I couldn't test it against the live archive. The smallest fix is to drop collapse (the regex filter already keeps the answer small) or to use collapse=urlkey-free paging.
  • .github/workflows/archive-resolve.yml (timeout-minutes: 20 on "Resolve archive links"), judgement call. The stop-early rule only covers refusals. A slow archive that still answers can use the full (CONNECT_TIMEOUT, TIMEOUT) = (10, 60) per request. At 146 requests plus the 1 s delay, that can pass 20 minutes. The step would then be killed before main() writes any YAML, and every replacement found so far would be lost until the next run. One fix: give resolve() a wall-clock budget that defers the rest, the same way a refusal does.
  • utils/archive_resolve.py (request_all), possible performance smell. The per-host semaphore is taken inside the pool worker. A run of consecutive URLs on one host (66 on pydata.org, 52 on pretalx.com in today's data) can hold up to 16 workers waiting on 2 slots. With 1,325 live URLs and a 5 s timeout this stays within budget, so it's not a bug. Also, the 10→5 s probe timeout turns more slow sites into timeout. That is correctly not broken, but those sites also drop out of redirect detection.

5. Tests

  • The new tests come from the request's invariants:

    • TestTurnedAway covers both the CDX and resolve sides: stop at the first refusal, no sleep after it, and a 503 doesn't stop the run.
    • TestRequestAll checks that every URL gets an answer and that no host exceeds PER_HOST.
    • TestNeedsList checks the cap.
    • TestLookupPlan covers shared hosts and dedupe.
    • TestHostCaptures / TestPageKey cover the request shape and the regex's keep/drop sets.

    The CDX tests use canned responses answers, so the real archive's filter and collapse semantics aren't exercised (see the first Nit).

  • Base vs change, same new test files (tests/test_archive_resolve.py tests/test_wayback.py):

    • base origin/main with the PR's test files: 27 failed, 76 passed in 0.52s (for example FAILED tests/test_wayback.py::TestHostCaptures::test_refusal_raises_blocked; the functions don't exist on base)
    • change: 103 passed in 0.29s
  • Full suite on the change (REVIEW.md uv line, -x): 2659 passed, 4 skipped, 3 xfailed, 2 xpassed, 251 warnings in 118.68s (0:01:58)

  • uvx ruff@0.4.4 check utils tests: All checks passed!

  • pre-commit.ci - pr: success checks completed successfully

6. Bounds

  • How I got the change: cloned the repo and fetched refs/pull/540/head; rev-parse matches ac34746. The diff and PR came from the REST API.
  • Read in full: the diffs of utils/archive_resolve.py, utils/wayback.py, utils/tidy_conf/links.py and the workflow, plus Probe.broken, superseded_candidates and site_host on the head.
  • Skimmed: the test diffs.
  • Not run:
    • The workflow itself.
    • Any live CDX query. web.archive.org is blocked here, so neither the regex filter nor the collapse behaviour was checked against the real archive.
    • voice-docs/Vale on the changed docstrings.
  • sort_yaml.py not run: no _data change.
  • Pre-existing: none noticed.

Generated by Claude Code

…urrency group

The 15 November run's refusals came in waves: runs of refusals with
successes after them, not one long block. Stopping at the first refusal
gave up the lookups after each wave.

- CDX requests go out one at a time, 1 s apart, with 5 s instead after
  every 10th (--spacing, --delay).
- A refused request is retried after 10, 20 and 40 s; still refused,
  the run stops asking and leaves the rest for the next run. Each
  refusal starts the waits again, and all backoff in one run is capped
  at 180 s, so a bad week costs at most 3 minutes of waiting.
- main() now passes --spacing through to resolve().
- archive-new-urls.yml and archive-resolve.yml share a job-level
  `wayback` concurrency group, so the two never send the archive
  requests at the same time from two runners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt
@JesperDramsch JesperDramsch changed the title Stop archive lookups at the first refusal, and check links in threads Pace archive lookups, back off on refusals, and check links in threads Nov 19, 2026
- wayback.host_captures: drop collapse=timestamp:8. CDX collapses
  adjacent lines, and in a whole-host answer one page's last capture
  sits next to the next page's first, so a page whose only capture fell
  on a day another page was captured would read as having none.
- archive_resolve.resolve: stop the lookups after TIME_BUDGET (600 s)
  and leave the rest for the next run, as after a refusal. A slow
  archive that still answers is never refused, and without this the
  step could hit its timeout before writing any replacement.
- archive_resolve.request_all: hand the URLs to the threads round robin
  over the hosts, so the workers do not all queue on the two slots of
  one big host such as pydata.org or pretalx.com.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt

Copy link
Copy Markdown
Owner Author

Replies to the routine review. It was written against ac34746. Since then a0519e9 replaced "stop at the first refusal" with pacing and backoff: 1 s between requests, 5 s after every 10th, then 10/20/40 s retries with at most 180 s of waiting per run. All three nits are fixed in 1aeaa8c.

  • collapse=timestamp:8 in host_captures (fixed). Agreed. CDX collapses adjacent lines, so in a whole-host answer one page's last capture can hide the next page's first capture of the same day. collapse is gone. The regex filter already keeps the answer down to the pages asked for. TestHostCaptures now asserts it isn't sent.
  • Step timeout against a slow archive (fixed). resolve() now has TIME_BUDGET (600 s). Once the lookups have used it, the rest is deferred to the next run, the same as after a refusal, and main() still writes what it found. The 20-minute step limit leaves room for the link check on top of that. Covered by TestTimeBudget.
  • Workers queuing on one host's slots (fixed). request_all now hands URLs to the threads round-robin across hosts, so a long run of pydata.org or pretalx.com URLs no longer holds most of the workers on two slots. Covered by TestRequestAll.test_hosts_are_interleaved.
  • Slow sites dropping out of redirect detection (no change). A site that times out at 5 s gives no final URL, so it can't be checked for a redirect that week. It isn't marked broken either, so nothing in the data changes, and it's checked again the following week. I'd rather keep the shorter timeout for the minutes it saves.

Full suite: 2664 passed, 4 skipped, 3 xfailed, 2 xpassed. pre-commit is clean.


Generated by Claude Code

@JesperDramsch JesperDramsch added the Review Ready Runs the PR review Routine; the Routine removes it after posting label Nov 19, 2026

Copy link
Copy Markdown
Owner Author

Routine review

1. Request

This is a re-review. The PR body is the request, and no issue is linked. Since the first review, the body has been rewritten. It now asks for these changes:

  • utils/wayback.py: per-host CDX lookups (host_captures(), page_key(), SHARED_HOSTS).
  • utils/archive_resolve.py: requests one at a time, 1 s apart, with 5 s after every 10th (--spacing, --delay). On a refusal (WaybackBlockedError), retry after 10/20/40 s, then stop. The backoff restarts on each new refusal, and all backoff waits are capped at 180 s per run.
  • A shared job-level wayback concurrency group in archive-new-urls.yml and archive-resolve.yml.
  • NEEDS:archive capped at 50 no-capture links.
  • A threaded link check: 16 workers, 2 per host, 5 s timeout, 30 min job cap, PYTHONUNBUFFERED, and a counts summary.

2. Checked against

  • Date: Thu Nov 19 10:51:21 UTC 2026
  • unverified: GitHub's behaviour for a concurrency group with cancel-in-progress: false: one running job and one pending job, with a newer pending job replacing the older one. The workflow comments and the PR body both state this. I did not fetch the docs in this review.
  • unverified, carried over from the first review: the PR body's live pycon.ru CDX check, "146 requests for 185 URLs", and the 65,536-character PR body limit. web.archive.org is blocked from this sandbox.

3. Needs your decision

None. The new work in a0519e94 and 1aeaa8c7 is named in the rewritten PR body: the backoff instead of stopping at the first refusal, the batch pacing, and the shared wayback group with its "a replaced resolve run waits a week" trade-off.

4. Findings

Earlier Important findings: the first review raised none, so nothing needed resolving.

New Important: None. I read the diff since ac34746 in full.

  • In resolve(), the retry loop tries the same plan item again after each wait in BACKOFF. It stops when the waits run out or when the next wait would pass backoff_budget. A plain WaybackError marks the item's URLs as failed, and the run moves on.
  • time_budget is checked before each plan item.
  • The batch pause is delay when i % batch_size == 0, and i is never 0 there. So the pause comes after every 10th request, as the request asks.
  • request_all now keys its result on the round-robin order. That holds the same URLs as before, so callers still find every key.

Following the re-review rule, there are no Nits. Commit 1aeaa8c7 does address the three Nits from the first review:

  • collapse is removed.
  • TIME_BUDGET = 600 s now defers the rest, the same way a refusal does.
  • Hosts are interleaved in request_all.

5. Tests

  • The new tests check the request's rules rather than fixed values:

    • TestTurnedAway: 10/20/40 s then stop, carrying on once a refusal lifts, a fresh backoff for each refusal, the budget cap, and a 503 that doesn't stop the run.
    • TestTimeBudget, TestBatches, and TestRequestAll::test_hosts_are_interleaved.
    • TestHostCaptures now asserts that collapse is absent.

    These tests use canned answers, so they don't exercise the real archive's semantics.

  • The PR's test files (tests/test_archive_resolve.py tests/test_wayback.py), run against base and change:

    • base origin/main: 37 failed, 71 passed in 0.76s (for example FAILED tests/test_wayback.py::TestHostCaptures::test_refusal_raises_blocked; the functions don't exist on base)
    • change: 108 passed in 0.37s
  • Full suite on the change (REVIEW.md uv line, -x): 2664 passed, 4 skipped, 3 xfailed, 2 xpassed, 251 warnings in 135.10s (0:02:15)

  • uvx ruff@0.4.4 check utils tests: All checks passed!

  • pre-commit.ci - pr on 1aeaa8c: success checks completed successfully

6. Bounds

  • How I got the change: cloned the repo and fetched refs/pull/540/head; rev-parse matches 1aeaa8c7c206422653e88cfb96b71efb860ac406. The PR, its commits and the earlier review came from the REST API.
  • Read in full:
    • git diff ac34746..HEAD for utils/ and .github/.
    • host_captures, page_key and _original_pattern on the head.
    • The archive-resolve.yml job and its steps, and the archive-new-urls.yml job header.
  • Skimmed: the test diff since ac34746. The earlier part of the PR was reviewed in the first round and I did not re-read it.
  • Not run:
    • The workflows.
    • Any live CDX query. web.archive.org is blocked here.
    • voice-docs/Vale on the changed docstrings and comments.
  • sort_yaml.py not run: no _data change.
  • Pre-existing: none noticed.

Generated by Claude Code

@JesperDramsch JesperDramsch removed the Review Ready Runs the PR review Routine; the Routine removes it after posting label Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants