skills.rest
brainbrewlabs
brainbrewlabsOFFICIAL
Explore all skills by this author →
GitHub

data-collection

Collect data from APIs, databases, and files into validated CSV, JSON, or Parquet outputs.

7|3|Updated Mar 17, 2026
Search Tags:#database#api#etl#csv#parquet#data-collection#data-ingestion
One-click install
npx skills add https://github.com/brainbrewlabs/brainbrew-devkit --skill data-collection
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-collection
Source: https://github.com/brainbrewlabs/brainbrew-devkit/tree/main/plugin/config/templates/data/skills/data-collection
Command: npx skills add https://github.com/brainbrewlabs/brainbrew-devkit --skill data-collection

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill automates extracting and centralizing structured data from disparate sources so downstream workflows can operate on clean, consolidated datasets without manual retrieval.

Core Features & Use Cases

  • Multi-source extraction: Connects to APIs, databases, and local files to pull raw data into a single pipeline.
  • Connector robustness: Handles authentication, pagination, rate limits, and common connection failures.
  • Structured output and validation: Writes outputs in CSV, JSON, or Parquet, validates schemas and record counts, and produces a summary of sources, record counts, and errors.
  • Use Case: Ingest transaction data from an API, merge with a database export, and produce a validated Parquet file for analytics.

Quick Start

Ask the skill to collect data from the provided API endpoint, handle authentication and pagination, validate the schema, and save the consolidated output as a CSV file.

Frequently Asked Questions about data-collection

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I extract data from an API and a database into a single dataset?▼

To collect data from APIs and databases into a unified dataset, the Skill handles authentication, pagination, and schema validation. It merges outputs from multiple sources into a validated CSV, JSON, or Parquet file.

What is the best way to handle API pagination and rate limits during data ingestion?▼

Data ingestion relies on built-in connector robustness to handle API pagination, rate limits, and common connection failures automatically. This ensures complete data extraction without manual intervention for interrupted requests.

Can I parse CSV and Parquet files together with database exports for ETL batch extraction?▼

Yes, ETL batch extraction supports parsing local CSV, JSON, and Parquet files alongside database exports. It consolidates these disparate sources into a single structured output while validating schemas and record counts.

How does schema validation work when collecting data from multiple sources?▼

Schema validation during data collection checks consolidated records against expected structures and counts. It ensures the unified dataset maintains integrity across all API, database, and file sources before outputting the final file.

Do I need to install CLI database clients to extract data using this Skill?▼

Yes, extracting data from databases requires CLI database clients as the Skill utilizes them to pull raw data. You must have the necessary client environments configured for the database connection to succeed.

What output formats are supported for consolidated data extraction?▼

Consolidated data extraction outputs validated datasets in CSV, JSON, or Parquet formats. Alongside the structured file, it produces a collection summary detailing sources, record counts, and any extraction errors encountered.

Related Skills

Explore More Data & Analytics Skills→
JuanMS20JuanMS20

github-auth

Configure GitHub authentication for Hermes Agent using git, gh CLI, or SSH credentials.

Community
Intermediate
starlake-aistarlake-ai
209

parquet2csv

Convert Parquet files into CSV format for batch ETL pipelines.

Official
Intermediate
andikarachmanandikarachman
14

data-preprocessing

Preprocess raw data into clean, analysis-ready datasets with pandas and numpy.

Community
Advanced
Nir-BhayNir-Bhay
19

airbyte-connection-setup

Configure Airbyte ETL/ELT pipelines and connectors via API.

Community
Advanced