Recommend CSV, Parquet, Feather, JSON, or Excel for data use cases.
npx skills add https://github.com/LeandroBenjaminL/lend-ai --skill file-formats
Please help me install this Agent Skill. Skill: file-formats Source: https://github.com/LeandroBenjaminL/lend-ai/tree/main/skills/file-formats Command: npx skills add https://github.com/LeandroBenjaminL/lend-ai --skill file-formats
This Skill provides a guide for selecting the most suitable file format for reading, writing, and converting data, addressing common issues like memory constraints and compatibility needs.
To determine the optimal file format for your data, run the file-formats skill and provide details about your use case, data volume, and target users.
High-intent search queries and answers about installing and using this skill.
Parquet is the best file format for large datasets that exceed memory limits in pandas, as it reduces memory footprint while retaining data compatibility and improving read and write performance.
You can convert a CSV file to Parquet using pandas by reading the CSV into a DataFrame and then writing it out to the Parquet format, which optimizes data storage and improves performance for large volumes.
Choose CSV for maximum data compatibility across platforms, Parquet for optimized performance and memory efficiency with large volumes, and Feather for fast read and write operations in pandas workflows.
You need to convert data formats for performance optimization when handling large data volumes that cause memory constraints, require faster reading and writing speeds, or need specific compatibility for target users.
Yes, pandas supports converting data between JSON, Excel, Feather, Parquet, and CSV formats, allowing you to optimize data manipulation capabilities based on your specific use case and performance requirements.
CSV limitations for large-scale data analysis in pandas include high memory consumption and slow reading and writing speeds, making formats like Parquet or Feather better alternatives for performance optimization.
datactivistAccess and process French open data from data.gouv.fr into pandas DataFrames.
eysenfalkOptimize AgentDB performance with quantization and HNSW indexing.
LeandroBenjaminLProfile datasets and validate schemas using pandas, ydata-profiling, and pandera.
youyinnnQuery and download public cancer imaging data from the National Cancer Institute Imaging Data Commons.
