LIVE · running

Production: scrape everything, or log exactly why not

The real worker needs every requested URL accounted for — either scraped, or in an error log with the reason and what was tried. This run targets 3,000 URLs/day through the full reliability pipeline and reports how many it gets, and what (if anything) it couldn't.

How it works

A captcha-walled URL isn't dropped. It runs the gauntlet — and only what's genuinely impossible (a dead link, a region outage) ends up in the log.

1

Rotate to a fresh exit

Most failures are a hostile Tor exit, not a bad URL. Rotate to a new exit and retry — ~99% of exits are clean, so this alone recovers nearly everything.

2

Escalate the solver

If a captcha won't clear, swap in a stronger model for the next round — qwen3-vl-32b → gemini-3.1-flash-lite. A URL isn't abandoned because one model missed.

3

Deferred retry pass

After the main sweep, the still-failed URLs are re-run on fresh exits for a few rounds — a URL that failed on a bad exit usually loads fine on a clean one.

4

Structured error log

Whatever's still unscraped gets one record: {url, category, attempts, what-was-tried} — shaped to drop straight into the worker's error handling.

By day