Generate PySpark and Spark SQL code for data processing and MLlib workflows.
npx skills add https://github.com/inbharatai/claude-skills --skill spark-builder
Please help me install this Agent Skill. Skill: spark-builder Source: https://github.com/inbharatai/claude-skills/tree/main/skills/spark-builder Command: npx skills add https://github.com/inbharatai/claude-skills --skill spark-builder
This Skill simplifies the creation and management of PySpark and Spark SQL jobs, enabling users to efficiently develop data processing and analytics pipelines.
Use the spark-builder skill to generate a PySpark script for reading a CSV file into a DataFrame.
High-intent search queries and answers about installing and using this skill.
PySpark streaming code is generated to process real-time data pipelines, such as reading from Kafka, performing transformations, and writing outputs to a data lake. The generated scripts handle RDDs, DataFrames, and streaming operations.
Spark SQL jobs are built by generating code that processes DataFrames and RDDs according to data analytics best practices. This provides the structured query logic needed to transform and analyze large datasets efficiently.
Yes, Spark cluster configuration is supported to optimize performance for machine learning tasks. It integrates with Spark's MLlib library to develop models while ensuring the underlying cluster is tuned for the workload.
Yes, Python and PySpark are required dependencies to execute the generated Spark jobs. The environment must have these installed to run the data processing, streaming, and MLlib scripts locally or on a cluster.
This approach accelerates big data processing workflows by generating PySpark and Spark SQL code that adheres to best practices. It simplifies creating data pipelines, reducing manual coding efforts for complex transformations.
Spark SQL may not be ideal for low-latency, real-time transactions outside big data batch or streaming contexts. It is designed for large-scale data processing and analytics pipelines rather than simple operational queries.
aryan083Automate creation and management of Spark jobs for distributed data processing.
clawdataAutomate building and running Apache Spark batch and streaming data processing jobs.
AbsolutelySkilledGuide building and operating Kafka and Flink streaming pipelines with exactly-once semantics.
silvainfmRun SQL analytics directly on CSV, Parquet, and JSON files and Pandas or Polars DataFrames.
