Extract municipal government data from Legistar and Municode using Scrapy spiders.
npx skills add https://github.com/stars-end/affordabot --skill context-scrapers
Please help me install this Agent Skill. Skill: context-scrapers Source: https://github.com/stars-end/affordabot/tree/main/.claude/skills/context-scrapers Command: npx skills add https://github.com/stars-end/affordabot --skill context-scrapers
This Skill provides a Scrapy-based scraping framework to collect data from Legistar city councils and Municode, with pipelines and verification scripts to ensure data quality and consistency.
Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion. For example, run scripts/verify_raw_scrapes.py to check the raw_scrapes table and verify_municode_discovery.py to validate Municode discovery.
High-intent search queries and answers about installing and using this skill.
Use Scrapy spiders to extract meetings and codes from Legistar and Municode. This Skill provides pre-built spiders, configurable pipelines for data storage, and Scrapy settings tailored to municipal sources, handling jurisdiction-specific variations and ensuring consistent ingestion.
Yes. Modify spider configuration and settings to target additional jurisdictions. The Skill includes verification scripts—verify_raw_scrapes.py and verify_municode_discovery.py—to validate that new jurisdictions ingest correctly and discover data reliably.
Run the included verification scripts to validate raw scrapes and discovery logic before full ingestion. These tools check data integrity, identify pipeline failures, and confirm that Legistar meetings and Municode codes are captured and stored consistently.
Web scraping against government sources requires validation. This Skill provides pipelines and verification scripts that check raw data consistency, detect schema drift, and confirm discovery logic across Legistar and Municode to catch ingestion issues early.
Yes. Scrapy settings control spider behavior, rate limiting, and storage paths per jurisdiction. This Skill bundles jurisdiction-specific configurations and pipelines, so you adjust settings rather than rewriting spiders when adding new cities or councils.
Legistar spiders extract meeting records, agendas, and votes; Municode spiders extract municipal code sections and ordinances. Both require different selectors and data schemas. This Skill provides separate spiders and pipelines tuned to each source's structure and update frequency.
gizixGenerate Scrapy spider templates for static, dynamic, and API content.
MichaelYang-lyxExtract classification distributions, clean text, and compute cross-tabulated statistics for analysis reports.
daffy0208Automate batch and real-time ETL pipelines with schema validation and monitoring.
chestnut0698Extract data from static and dynamic web pages using Python tools.
