Create Parquet files from C# data sources with incremental updates.
npx skills add https://github.com/lawless-m/claude-skills --skill parquet-files
Please help me install this Agent Skill. Skill: Parquet Files Source: https://github.com/lawless-m/claude-skills/tree/main/Parquet Files Command: npx skills add https://github.com/lawless-m/claude-skills --skill parquet-files
What problem does it solves? This Skill streamlines the creation, management, and incremental updating of Parquet files in C#, addressing challenges like memory management, multi-threading, and dynamic schema generation for large datasets. It helps users efficiently store and process big data, saving development time and optimizing resource usage.
DataTable or dictionary structures, handling type conversions.ParquetUpdateQueue and SemaphoreSlim for safe, concurrent writing and incremental updates to Parquet files.Use the Parquet Files skill to help me create a Parquet file from a C# DataTable, ensuring efficient memory usage for large datasets.
High-intent search queries and answers about installing and using this skill.
Create Parquet files by using Parquet.Net to dynamically generate schemas from your DataTable, process data in batches, and implement explicit garbage collection between batches to prevent OutOfMemory exceptions when handling large datasets.
Yes, use ParquetUpdateQueue for coordination and SemaphoreSlim for serialized writes to safely append new records to Parquet files concurrently, ensuring thread-safe incremental updates without data corruption.
Build a data pipeline using Parquet.Net with batch sizing and multi-threaded writing via ConcurrentBag<T> for data collection, implementing incremental updates and memory-efficient processing to continuously sync and deduplicate large MySQL datasets.
Parquet.Net automatically generates Parquet schemas from dictionary structures and DataTables with type conversion, enabling you to export data from external APIs and Elasticsearch without manually defining schemas.
This Skill requires Parquet.Net versions 4.23.5 through 4.25.0; these versions provide the API stability and memory handling needed for multi-threaded batch writing and garbage collection coordination in production pipelines.
Implement batch-level logging and explicit memory management with GC.Collect() and GC.WaitForPendingFinalizers() after each batch write, combined with ConcurrentBag<T> clearing to keep heap usage predictable across multi-threaded operations.
bcmcpherGenerate a JSON data dictionary describing variables in merged CSV, TSV, or Parquet datasets.
Alex1980AlexAnalyze JSONL and Parquet logs directly with DuckDB.
dkyazzentwatwaTransform tabular data into narrative reports with automated insights and visualizations.
lucasfdigitalBuild data pipelines with Polars, DuckDB, Apache Arrow, and dbt.