Creates SparkSessions and performs PySpark ETL with CSV, Parquet, JSON, and Delta.
npx skills add https://github.com/timequity/plugins --skill spark-basics
Please help me install this Agent Skill. Skill: spark-basics Source: https://github.com/timequity/plugins/tree/main/craft-coder/data/spark-basics Command: npx skills add https://github.com/timequity/plugins --skill spark-basics
PySpark basics for distributed data processing, enabling scalable ETL.
Create a SparkSession and run a few simple reads and transforms on a sample dataset.
High-intent search queries and answers about installing and using this skill.
PySpark enables distributed ETL by creating a SparkSession, reading data from CSV, Parquet, JSON, or Delta Lake formats, applying transformations like select and filter across partitions, then writing results back. This scales data processing across clusters for analytics tasks.
SparkSession handles diverse formats—CSV, Parquet, JSON, Delta—through unified read APIs. Chain transformations like select, filter, and aggregate on the resulting DataFrames, then write to Parquet or Delta Lake with optimization strategies like caching and predicate pushdown.
Optimization involves enabling adaptive execution in SparkSession, using caching for frequently accessed DataFrames, applying predicate pushdown to filter early, and leveraging partitioning and joins. These techniques reduce shuffles and memory usage across distributed executors.
Yes. PySpark supports aggregations, joins across DataFrames, and window functions for analytics. These operations work on partitioned data across clusters, making them suitable for large-scale transformations before writing to Parquet or Delta Lake.
Create a SparkSession as your entry point to the Spark cluster. Prepare data in supported formats like CSV, Parquet, JSON, or Delta Lake, and ensure your source (S3, local, or other storage) is accessible. Then read, transform, and write using DataFrame operations.
handsomelong922Scale distributed data processing for ML pipelines across clusters.
Awish021Build and optimize PySpark ETL pipelines on Databricks with Delta Lake.
4lermanBuild scalable data pipelines with Python, SQL, Spark, and Airflow.
aselimcScale pandas and NumPy workflows to larger-than-memory datasets with parallel and distributed computing.
