Repository navigation
Pace archive lookups, back off on refusals, and check links in threads - #540
JesperDramsch wants to merge 3 commits into
Conversation
The first Archive Resolve run (15 Nov) replaced 65 links, then the archive started refusing connections: 275 of the 301 links it kept were failed lookups, and the run kept asking one a second through all of them. It also ended red because the list of kept links made the rolling PR body 165,880 characters, over GitHub's 65,536. - wayback: a refused connection or HTTP 429/509 on a CDX request raises WaybackBlockedError. host_captures() lists several pages of one host in a single request, with a filter on the archive's side so the answer holds only those pages; page_key() matches captures to the data as CDX does (no www., scheme, trailing slash or #fragment). - archive_resolve: pages of a host with several candidates share one request; hosts that carry many sites (SHARED_HOSTS) stay one page per request. At the first refusal the run stops asking, without waiting, and leaves the rest for the next run, which finds them again. NEEDS:archive lists only links with no capture from their edition's time, at most 50; failed and deferred lookups are retried, not listed. - The link check requests URLs in 16 threads of the one runner, at most two at a time per site, with a 5 s timeout instead of 10 s. - The workflow prints as it runs, writes the run's counts to the job summary, and caps the job at 30 minutes instead of 180. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt
|
Routine review 1. RequestThe PR body is the request, and no issue is linked. It asks for four changes after the 15 November Archive Resolve run:
2. Checked against
3. Needs your decisionNone. Everything the diff does is named in the PR body. 4. FindingsImportant: None. Nits:
5. Tests
6. Bounds
Generated by Claude Code |
…urrency group The 15 November run's refusals came in waves: runs of refusals with successes after them, not one long block. Stopping at the first refusal gave up the lookups after each wave. - CDX requests go out one at a time, 1 s apart, with 5 s instead after every 10th (--spacing, --delay). - A refused request is retried after 10, 20 and 40 s; still refused, the run stops asking and leaves the rest for the next run. Each refusal starts the waits again, and all backoff in one run is capped at 180 s, so a bad week costs at most 3 minutes of waiting. - main() now passes --spacing through to resolve(). - archive-new-urls.yml and archive-resolve.yml share a job-level `wayback` concurrency group, so the two never send the archive requests at the same time from two runners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt
- wayback.host_captures: drop collapse=timestamp:8. CDX collapses adjacent lines, and in a whole-host answer one page's last capture sits next to the next page's first, so a page whose only capture fell on a day another page was captured would read as having none. - archive_resolve.resolve: stop the lookups after TIME_BUDGET (600 s) and leave the rest for the next run, as after a refusal. A slow archive that still answers is never refused, and without this the step could hit its timeout before writing any replacement. - archive_resolve.request_all: hand the URLs to the threads round robin over the hosts, so the workers do not all queue on the two slots of one big host such as pydata.org or pretalx.com. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt
|
Replies to the routine review. It was written against
Full suite: 2664 passed, 4 skipped, 3 xfailed, 2 xpassed. pre-commit is clean. Generated by Claude Code |
|
Routine review 1. RequestThis is a re-review. The PR body is the request, and no issue is linked. Since the first review, the body has been rewritten. It now asks for these changes:
2. Checked against
3. Needs your decisionNone. The new work in 4. FindingsEarlier Important findings: the first review raised none, so nothing needed resolving. New Important: None. I read the diff since
Following the re-review rule, there are no Nits. Commit
5. Tests
6. Bounds
Generated by Claude Code |
Why
The first Archive Resolve run (15 November, run 43705009939) found 270 yearless URLs on superseded editions and 127 broken ones. It replaced 65 of them, and those reached
mainthrough #535. Then the archive started refusing connections. Of the 301 links it left in place, 275 were failed lookups and only 26 had no capture. The refusals came in waves: about 20 successes, then runs of 20 or more refusals, each followed by more successes. The run kept sending a request every second through all of them. It also ended red: the list of kept links would have made the rolling PR body 165,880 characters, and GitHub's limit is 65,536. It took 38 minutes of Actions time.What changes
Fewer CDX requests (
utils/wayback.py).host_captures()lists the captures of several pages on one host in a single request,url=<host>/*. A filter on the archive's side (filter=original:<regex>) keeps the answer to just those pages. Checked by hand againstpycon.ru: the filtered answer holds only/,/cfpand/partneram, including thewww.variants, while the unfiltered one lists speaker pages,robots.txtand a sitemap.collapse. CDX collapses adjacent lines, so in a whole-host answer one page's capture could hide another page's capture from the same day.page_key()matches captures to the data the way CDX files them: withoutwww., scheme, trailing slash or#fragment.SHARED_HOSTS: Google Forms and Drive,pydata.org, pretalx, Sessionize, GitHub, Linux Foundation and others) stay one page per request.Pacing, backoff and a time limit (
utils/archive_resolve.py).--spacing,--delay).WaybackBlockedError, the CDX counterpart of what Stop the archive save run when the Wayback Machine turns it away #539 does for saves. The same request is retried after 10, 20 and 40 s. If it's still refused, the run stops asking, and the next run finds the remaining links again.TIME_BUDGET) and leave the rest for the next run, the same as after a refusal. A slow archive that still answers can't run the step into its timeout and lose the replacements already found.Only one Wayback job at a time.
archive-new-urls.ymlandarchive-resolve.ymlshare a job-levelwaybackconcurrency group, so they never send the archive requests at once from two runners. A pending job can be replaced by a newer one. For the save job that loses nothing, because it starts from its last successful run. A replaced resolve run waits until the next week.NEEDS:archiveonly lists what needs a person. It holds the links with no capture from their edition's time, 50 at most, with a count of the rest. Failed and deferred lookups aren't listed, because the next run retries them. This keeps the body under GitHub's limit.Fewer Actions minutes in the link check.
PYTHONUNBUFFERED=1makes the log show where the time goes.Tests
TestLookupPlan,TestResolveByHost,TestTurnedAway,TestTimeBudget,TestBatches,TestNeedsList,TestRequestAll,TestHostCaptures,TestPageKey, and theWaybackBlockedErrorcases for CDX.TestTurnedAwaycovers backing off 10/20/40 s then stopping, carrying on once a refusal lifts, restarting the backoff for each refusal, the 180 s cap, and other failures not stopping the run.TestBatcheschecks 1 s within a batch and 5 s after every 10th request.TestRequestAllchecks every URL gets answered, at mostPER_HOSTrequests run per site at once, and hosts are interleaved.pytest tests/(REVIEW.md's uv line): 2664 passed, 4 skipped, 3 xfailed, 2 xpassed.Not run: the workflows themselves. They run next on their crons, or by hand with
workflow_dispatchonce this is merged.🤖 Generated with Claude Code
https://claude.ai/code/session_016a2XGgiRrtWm4EUP7ek9wt