skills.rest
tcole333
tcole333COMMUNITY
Explore all skills by this author →
GitHub

ingest-source

Automate data source onboarding with schema analysis and query wrapper generation.

Updated Feb 18, 2026
Search Tags:#data ingestion#data schema#api endpoint#investigative tools#source characterization
One-click install
npx skills add https://github.com/tcole333/ithildin --skill ingest-source
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: ingest-source
Source: https://github.com/tcole333/ithildin/tree/main/.claude/skills/ingest-source
Command: npx skills add https://github.com/tcole333/ithildin --skill ingest-source

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires datasets, requests, pandas, sqlite3, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Streamline the process of onboarding new data sources into investigative platforms, enhancing data integration capabilities.

Core Features & Use Cases

  • Source Characterization: Automate characterization of data sources (format, size, content, provenance, overlap).
  • Data Download & Access: Handle direct downloads and API data retrieval with endpoint discovery and documentation.
  • Schema Analysis: Analyze and output data schema details like columns, data types, and row count.
  • Query Wrapper Creation: Facilitate creation of query wrappers for custom data source queries.
  • Initial Investigation: Perform initial searches to identify relevant data from new sources.
  • Lead Generation: Generate leads based on significant findings in new data sources.
  • Source Reporting: Update source reporting tools with new data sources.
  • CLAUDE.md Integration: Include new data source information in the platform's inventory guide.
  • Ingestion Logging: Document data ingestion details for future reference.

Quick Start

Use the ingest-source skill to onboard the new data source 'tensonaut/EPSTEIN_FILES_20K'.

Frequently Asked Questions about ingest-source

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate data source onboarding for an investigative platform?▼

Automating data source onboarding involves characterizing the source format, discovering API endpoints, and analyzing the data schema. This process streamlines integration by generating query wrappers and logging ingestion details for investigative platforms.

What does data source characterization involve when integrating a new API endpoint?▼

Data source characterization involves identifying the format, size, content, provenance, and overlap of the dataset. This analysis is necessary to understand the structure and relevance of a new API endpoint before integrating it into an investigative platform.

Can I use pandas and sqlite3 to analyze data schema and generate query wrappers?▼

Yes, pandas and sqlite3 can analyze data schema details like columns and data types to facilitate query wrapper creation. These dependencies support the initial data searching and logging required for robust investigative tool integration.

What is the best way to handle direct downloads and API data retrieval for investigation?▼

The best way to handle direct downloads and API data retrieval is by automating endpoint discovery and documentation. This approach ensures the data source is properly characterized and accessible for initial searches and lead generation.

Does the data ingestion process update the platform inventory guide automatically?▼

Yes, the data ingestion process updates the platform inventory guide through CLAUDE.md integration. It documents the ingestion details and updates source reporting tools to include the new data source information for future reference.

Related Skills

Explore More Software Engineering Skills→
angryracoonangryracoon

data360-prepare

Prepare Salesforce Data Cloud ingestion assets for raw source data.

Community
Advanced
apavanelloapavanello

researchers-journalism

Research and analyze journalistic sources for comprehensive reporting.

Community
Advanced
BirdywenBirdywen
1

sqlite-data

Create, query, analyze, and export SQLite data via the sqlite3 CLI.

Community
Intermediate
monte-carlo-datamonte-carlo-data
90

push-ingestion

Generate push ingestion scripts for Monte Carlo's Ingestion API across data warehouses.

Official
Advanced