skills.rest
UnicomAI
UnicomAICOMMUNITY
Explore all skills by this author →
GitHub

smart-data-collection

Extracts structured data from images and documents, then imports it into databases via the ontology CLI.

2.5k|134|Updated Jun 6, 2025
Search Tags:#ontology#OCR#etl#Database#document parsing#csv import#Data extraction
One-click install
npx skills add https://github.com/UnicomAI/wanwu --skill smart-data-collection
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: smart-data-collection
Source: https://github.com/UnicomAI/wanwu/tree/main/configs/microservice/bff-service/configs/agent-skills/ontology/smart-data-collection
Command: npx skills add https://github.com/UnicomAI/wanwu --skill smart-data-collection

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

It turns unstructured files like images, PDFs, Word, and Excel documents into structured database records without manual data entry, while enforcing safe, audited writes through the platform's data-flow channel instead of direct database connections.

Core Features & Use Cases

  • Document-to-CSV Parsing: Converts images (via OCR) and documents (PDF/Word/Excel) into Markdown tables, maps fields to knowledge-network object-type definitions, and produces RFC 4180 CSV files.
  • Safe Database Import: Resolves the write target through object-type, dataview, and datasource lookups, deduplicates against existing primary keys, and imports rows via ontology ds import-csv with batch control.
  • Guardrails and Confirmation: Blocks direct MySQL/JDBC writes, requires per-field user confirmation on mapping ambiguities, and demands explicit approval of the write plan before execution.
  • Use Case: A user uploads a scanned product list image; the skill OCRs it, maps columns to the product object type, removes rows whose primary keys already exist, and inserts only the new rows into the target table.

Quick Start

Ask the assistant to import your CSV or document into a specific object class, for example: import the file products.csv into the product object type of my knowledge network.

Frequently Asked Questions about smart-data-collection

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I import data from images or PDFs into a database?▼

Provide the image or document file and target object type. The skill parses the file into Markdown tables via OCR or document parsing, maps fields to the object-type definition with your confirmation, writes a CSV, and imports it through the ontology ds import-csv command.

How does the skill prevent duplicate primary key errors during import?▼

Before importing, it queries the target table for existing primary keys using dataview query, then removes matching rows from the CSV. Only the deduplicated rows are sent to ds import-csv, avoiding MySQL Error 1062 duplicate entry failures.

Can I update or overwrite existing rows with this import tool?▼

No. The ds import-csv channel only supports INSERT operations, not UPDATE, UPSERT, or REPLACE. Rows with existing primary keys are excluded before import, and modification requests are declined.

What file formats are supported for data extraction?▼

Supported inputs include images (.jpg, .jpeg, .png, .bmp), PDF, Word (.doc, .docx), and Excel (.xls, .xlsx) files, plus ready-to-use CSV files that skip the parsing phase entirely.

Why does the skill forbid direct MySQL connections for writing data?▼

Direct connections bypass platform auditing, permissions, field mapping, and index refresh, causing the knowledge-network index to diverge from the base table. All writes must go through the platform data-flow service via ds import-csv.

What happens when field mapping between my document and the object type is ambiguous?▼

The skill stops and presents a comparison table, asking you to decide each differing field one at a time: confirm the suggested mapping, choose another target field, skip the field, or terminate. No CSV is written until every discrepancy is resolved.

Related Skills

Explore More Data & Analytics Skills→
AdawoduAdawodu

dynosist

Compose Gmail drafts with attachments and search emails and files.

Community
Intermediate
kweaver-aikweaver-ai

create-bkn

Generate complete BKN definitions with YAML frontmatter for kweaver-core.

Official
Intermediate
sixerssixers

fakturownia-invoice-receipt-link

Link receipts to Fakturownia invoices or create invoices from receipt data.

Community
Basic
luthebaoluthebao
1

expo-dev-client

Build and submit custom Expo development clients to TestFlight via EAS Build.

Community
Intermediate