skills.rest
alongor666
alongor666COMMUNITY
Explore all skills by this author →
GitHub

backend-data-processor

Process vehicle insurance Excel/CSV data into clean, merged datasets.

1|Updated Nov 4, 2025
Search Tags:#backend#data pipeline#Excel#data processing#data cleaning#Pandas#CSV
One-click install
npx skills add https://github.com/alongor666/daylyreport --skill backend-data-processor
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: backend-data-processor
Source: https://github.com/alongor666/daylyreport/tree/main/.claude/skills/backend-data-processor
Command: npx skills add https://github.com/alongor666/daylyreport --skill backend-data-processor

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires pandas, openpyxl.

What problem does it solve?

This Skill provides a comprehensive guide for backend data processing of vehicle insurance Excel/CSV files using Pandas. It automates file handling, data cleaning, merging, and validation, ensuring data readiness for analysis and reporting.

Core Features & Use Cases

  • Data Flow Architecture: Outlines the complete process from Excel upload to merged CSV storage, including incremental updates and file versioning.
  • Core Processing Functions: Details methods for reading Excel, cleaning data (missing values, date/numeric standardization), merging with existing data, and batch processing new files.
  • Business Rule Implementation: Explains how to handle specific business rules like negative premiums (retained), zero commissions (retained), and date standardization.
  • Performance Optimization: Offers Pandas best practices for vectorized operations, memory management, and strategies for handling large files.
  • Use Case: A data engineer needs to integrate new daily Excel reports into the existing analytics platform. This skill guides them through setting up the data pipeline, ensuring new data is cleaned, deduplicated, and merged efficiently with historical records.

Quick Start

Process the new Excel file 'daily_insurance_report.xlsx' by cleaning, merging with existing data, and saving the updated dataset.

Frequently Asked Questions about backend-data-processor

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I clean and deduplicate Excel and CSV data with Pandas?▼

Data cleaning with Pandas involves reading Excel/CSV files, standardizing dates and numbers, removing duplicates by policy identifiers, and handling missing values. This Skill provides methods for these operations, including handling UTF-8-SIG encoding and retaining business rules like negative premiums and zero commissions.

Can I merge new Excel reports with existing datasets incrementally?▼

Yes. This Skill guides incremental merging where new Excel/CSV data is cleaned, deduplicated, and merged with historical records in a single pipeline. The process maintains data versioning and produces updated CSV exports ready for analysis.

What's the best way to process large Excel files without memory issues?▼

Pandas vectorized operations and memory-efficient batch processing are core techniques. This Skill covers file I/O optimization, large-file handling strategies, and performance best practices to process substantial datasets without excessive memory consumption.

How do I automate a data pipeline for daily insurance report ingestion?▼

This Skill outlines a complete data flow architecture from Excel upload through cleaning, deduplication, and merging with existing data to final CSV storage. It covers error management, logging, and batch processing for daily automated workflows.

Does Pandas handle date and number standardization across different Excel formats?▼

Yes. This Skill explains standardization methods for dates and numeric fields across varied Excel/CSV formats, including handling encoding variations and applying consistent business rules to ensure analysis-ready datasets.

What data validation should I apply before merging insurance records?▼

This Skill covers deduplication by policy number and confirmation time, validation of file structure, and error handling during the merge process. It ensures only clean, standardized records are combined with existing data to maintain dataset integrity.

Related Skills

Explore More Data & Analytics Skills→
zjy-devzjy-dev
2

xlsx

Create, edit, and analyze Excel files with pandas and openpyxl.

Community
Advanced
SJTU-IPADSSJTU-IPADS
7

data-analyst-cn

Clean datasets and generate statistics, correlations, and visualization templates.

Official
Intermediate
delta-and-betadelta-and-beta

engineer-cloud-ingestion

Configure cloud data ingestion connectors and transformation pipelines for Braze.

Community
Advanced
moshi-azzmoshi-azz
2

experto-excel

Guide Excel users through formulas, VBA macros, and data analysis.

Community
Advanced