fetching-and-extracting

fetchSmart tries a static got request first and renders with Playwright only if the page looks empty (fewer than 5 links or under 500 chars of body text). The needsJs result is cached per host so the decision is not re-made, and forceRender bypasses the heuristic.

A forked child process isolates native crashes: a JSDOM CSS-parser segfault in worker_threads kills the whole process because threads share memory, while a forked child contains it. On child crash the pool resolves the task to a safe default and respawns, never re-running inline.

The listing profile harvests link href, text, and datetime each scroll round and stops when at least 2 new dated items are all older than the --since floor, when 3 consecutive checks yield no new hrefs, or at the round/deadline cap. Harvested links merge conservatively into the crawl queue.

Link selectors must yield at least 3 unique links, content selectors at least 400 characters, and date specs must produce parseable dates for at least max(3, 50%) of items. Below threshold the selector is re-derived instead of cached.

Orphaned Chromium child processes with open pipes can hold the Node event loop after a run ends. The fix gives browser close 10 seconds, then SIGKILLs direct Chromium children of the process, with a CLI backstop timer that exits after the database closes.

isBlockedPage checks title and text for Cloudflare or captcha challenge patterns and gates every save, since challenge pages return HTTP 200. For curated items a blocked target keeps the aggregator blurb instead of being dropped.