driver-csv-ingest

Ingest CSV and TSV files into local feather storage with encoding detection.

157|31|Updated May 27, 2020

Search Tags:#automation #ingestion #csv #data storage #tsv #feather

Ready to run

npx skills add https://github.com/bbartling/open-fdd --skill driver-csv-ingest

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: driver-csv-ingest
Source: https://github.com/bbartling/open-fdd/tree/main/skills/driver-csv-ingest
Command: npx skills add https://github.com/bbartling/open-fdd --skill driver-csv-ingest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the process of importing CSV and TSV files into local feather storage systems, enabling efficient data analysis and management.

Core Features & Use Cases

File Upload Handling: Accepts multipart uploads or path-based CSV/TSV files for ingestion.
Encoding Detection and Normalization: Identifies file encoding formats such as UTF-16 LE BOM and tab separation, normalizing timestamps to ensure consistency.
Data Loading and Preview: Loads data into feather storage, providing row counts and sample previews for validation.
Use Case: Imagine a data engineer importing hourly sensor data stored as CSVs into a local database, then querying for recent metrics in Grafana.

Quick Start

Instruct the system to process a CSV file by uploading it to the specified route and verifying the ingestion success through the provided metrics.

Frequently Asked Questions about driver-csv-ingest

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I ingest CSV and TSV files into local storage for analysis?▼

To ingest CSV and TSV files into local storage, this Skill accepts multipart uploads or path-based files, automatically detects encoding formats like UTF-16 LE BOM, normalizes timestamps, and loads data into feather storage with row count and preview validation.

What is the best way to automate batch CSV ingestion with encoding detection?▼

Automating batch CSV ingestion with encoding detection is handled by this Skill through automatic identification of formats such as UTF-16 LE BOM and tab separation, normalizing timestamps for consistent batch data loading into local feather storage.

Does pandas support converting CSV to feather format with timestamp normalization?▼

Yes, this Skill uses pandas to convert CSV and TSV files into feather format, applying automatic encoding detection and timestamp normalization to ensure data consistency during local storage ingestion.

Can I validate data integrity after importing a TSV file into feather storage?▼

You can validate data integrity after importing a TSV file into feather storage because this Skill provides row count summaries and sample data previews immediately following the ingestion process.

How do I handle UTF-16 LE BOM encoded CSV files during data ingestion?▼

Handling UTF-16 LE BOM encoded CSV files during data ingestion is automatic with this Skill, which detects the encoding format alongside tab separation and normalizes the data before loading it into local feather storage.

Why does CSV ingestion fail when file encoding is not detected automatically?▼

CSV ingestion can fail without automatic encoding detection when formats like UTF-16 LE BOM are misinterpreted, which this Skill prevents by automatically identifying encoding and normalizing timestamps before loading data into feather storage.