Skills
.Work. You
Rest

hops-spark

Official

Effortlessly execute PySpark code in Hopsworks for distributed data processing.

Software Engineering#PySpark#Delta Lake#Hopsworks#Distributed processing#Data science
Authorlogicalclocks
Version1.0.0
Installs0

System Documentation

What problem does it solve?

This Skill unit simplifies the execution of PySpark code within the Hopsworks environment, addressing the need for distributed data processing that exceeds the capabilities of a single machine.

Core Features & Use Cases

  • PySpark Execution: Facilitates the execution of PySpark code in Hopsworks for large-scale data processing.
  • Spark Connect Mode: Automatically invokes Spark Connect mode for seamless integration with Hopsworks.
  • Delta Lake Integration: Ensures seamless interaction with Delta Lake for offline feature groups and data storage.
  • Use Case: Ideal for data scientists and engineers who need to process large datasets using PySpark within the Hopsworks platform.

Quick Start

To execute a PySpark script in Hopsworks, use the hopsworks SDK with the build_spark function, like so: spark = build_spark("my_pipeline").

Dependency Matrix

Required Modules

None required

Components

scripts

💻 Claude Code Installation

Recommended: Let Claude install automatically. Simply copy and paste the text below to Claude Code.

Please help me install this Skill:
Name: hops-spark
Download link: https://github.com/logicalclocks/hopsworks-api/archive/main.zip#hops-spark

Please download this .zip file, extract it, and install it in the .claude/skills/ directory.
View Source Repository

Agent Skills Search Helper

Install a tiny helper to your Agent, search and equip skill from 620,000+ vetted skills library on demand.