rj-scrape

Automating job scraping involves using a Patchright-managed browser to pull new job listings from configured sources and insert them directly into the rocket-jobs-app database. It eliminates manual scraping by detecting missing logins and re-running automatically after authentication.

Login detection works by using a persistent Chrome profile to auto-detect missing login states during the job scraping process. If a login is required, the process pauses and uses the /api/scrape endpoint to re-run the ingestion after authentication is completed.

Yes, you need the web app running locally at http://localhost:3000 with at least one source enabled in Settings. The server reads enabled sources directly from the app configuration, so the scraping skill never maintains the source list manually.

The best way to scrape job listings without managing login state is using a persistent Chrome profile that auto-detects missing authentication. This approach handles login detection automatically and re-runs the scraping endpoint once access is restored.

Automated job scraping fails to pull new listings if the local web app is not running at http://localhost:3000 or if no sources are enabled in Settings. Additionally, missing logins can halt the process until authentication is completed through the browser.

Yes, rj-scrape can pull listings from custom sources as long as they are configured and enabled in the rocket-jobs-app Settings. The skill reads the enabled sources directly from the server configuration to automate the scraping process.