gcp-crawler
CommunityBuild large-scale GCP corpus crawler, manage scraping & extracting workflows.
Authordloschiavo
Version1.0.0
Installs0
System Documentation
What problem does it solve?
This Skill automates large-scale corpus crawling on GCP using Cloud Run Jobs, simplifying scraping, caching, and extraction processes, saving time and effort.
Core Features & Use Cases
- Cloud Run Jobs: Run scrapers and extractors as long-running Cloud Run jobs, efficiently handling large-scale tasks.
- GCS Storage: Cache all raw bytes to Google Cloud Storage before processing, ensuring durability and resumability.
- Manifest-based Queue: Manage crawling workflows through a Cloud SQL Postgres manifest table, ensuring scalability and efficient resource allocation.
- Scraper and Extractor Management: Use predefined scrapers and extractors to handle web scraping and data extraction, with the ability to extend and customize.
Quick Start
Use the gcp-crawler skill to start scraping a new source from a list of URLs provided in 'urls.txt'.
Dependency Matrix
Required Modules
cloud-runcloud-sqlgcspostgres
Components
scriptsreferences
💻 Claude Code Installation
Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.
Please help me install this Skill: Name: gcp-crawler Download link: https://github.com/dloschiavo/orca/archive/main.zip#gcp-crawler Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
Agent Skills Search Helper
Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.