Manipulate large datasets with DuckDB-backed dplyr syntax.
npx skills add https://github.com/arthurgailes/awesome-r-skills --skill r-duckplyr
Please help me install this Agent Skill. Skill: r-duckplyr Source: https://github.com/arthurgailes/awesome-r-skills/tree/main/skills/r-duckplyr Command: npx skills add https://github.com/arthurgailes/awesome-r-skills --skill r-duckplyr
This Skill addresses the performance bottlenecks and memory limitations encountered when using dplyr with large datasets (over 100k rows) or larger-than-memory files, by leveraging DuckDB for efficient, lazy data processing.
dplyr syntax, performing complex filtering, grouping, and summarization without running out of memory.Use the r-duckplyr skill to read the file 'large_dataset.parquet' and filter rows where the 'year' column is 2024.
High-intent search queries and answers about installing and using this skill.
Process large datasets exceeding available RAM by using a DuckDB-backed implementation of dplyr syntax. This approach leverages lazy evaluation and queries disk-based files like Parquet and CSV directly, preventing memory overflow during data manipulation.
Yes, direct querying of Parquet, CSV, and other file formats from disk or URLs is supported without loading them entirely into RAM. This larger-than-memory support allows you to perform complex filtering and summarization on multi-gigabyte files efficiently.
Lazy evaluation in big data processing means operations are executed only when results are needed. This optimizes performance for large datasets by minimizing unnecessary memory usage and accelerating complex data manipulation pipelines.
Yes, r-duckplyr enables efficient data manipulation on large datasets using familiar dplyr syntax. It acts as a DuckDB backend, allowing you to perform complex filtering, grouping, and summarization without learning a new query language.
The best way to handle multi-gigabyte Parquet files is utilizing a DuckDB backend with lazy evaluation. This method allows you to execute complex grouping and summarization directly on disk-based files using standard dplyr verbs without running out of memory.
mrl2013Manipulate R data frames with dplyr for filtering, sorting, and summarizing.
kivo360Retrieve Git status, SPEC progress, version information, and system resources into a formatted status report.
ivyleavedtoadflaxGenerates R code for accessible spreadsheets, GOV.UK charts, and R Markdown reports using Analysis Function standards.
logicalclocksQuery Hopsworks feature store tables using Trino SQL.