Fetch, transform, and deduplicate incident data from external APIs.
npx skills add https://github.com/ohall/thesituation --skill data-collector
Please help me install this Agent Skill. Skill: data-collector Source: https://github.com/ohall/thesituation/tree/main/.skills/data-collector Command: npx skills add https://github.com/ohall/thesituation --skill data-collector
Pattern for building data source collectors that fetch, transform, and deduplicate incident data from external APIs. Use when adding new data sources, fixing collector bugs, or understanding the data pipeline.
Create a new collector by implementing the Collector interface, add it to the collectors registry, and run the processor to ingest and upsert incidents.
High-intent search queries and answers about installing and using this skill.
To build a resilient data collector, implement a standard Collector interface with fetch() and transform(raw) methods to fetch and normalize raw data into a unified Incident shape. This pattern provides guardrails like upsert to storage, dedupe keys, and error handling during fetch or transform.
The best way to deduplicate incident data is using a collector pattern that tracks dedupe keys during the transformation phase. The standard Collector interface normalizes raw API data into a unified Incident shape and upserts to storage, preventing duplicate records.
To add a new data source to an ETL pipeline, implement the Collector interface with fetch and transform methods, add it to the collectors registry, and run the processor to ingest and upsert incidents. Example implementations like NYC FDNY, 511NY, and NWS illustrate integration patterns.
Yes, you can normalize raw API data from different sources by implementing the transform(raw) method within the Collector interface. This method takes raw data from external APIs and converts it into a unified Incident shape before upserting to storage.
Data collectors can fail during fetch or transform due to unhandled errors from external APIs. The collector pattern includes guardrails for reliability and error handling during these stages, along with health tracking to monitor and debug collector workflows.
No external dependencies are required to implement the data collector pattern. It defines a standard Collector interface and registry system that can be run via a processor to ingest and upsert incidents without relying on additional packages.
abzhawAutomate REST API calls with retry, backoff, and authentication in Julia.
CAPHTECHDetect production law violations in telemetry and logs and feed results into pce-memory.
hieutrtrImplement logging, metrics, tracing, health checks, alerting, and dashboards for Python/React services.
pengfitAutomate ETL processes for government price data across 17 provinces and cities.
