skills.rest
dominodatalab
dominodatalabOFFICIAL
Explore all skills by this author →
WebsiteGitHub

domino-distributed-computing

Configure and launch Spark, Ray, and Dask clusters in Domino Data Lab.

6|3|Updated Feb 19, 2026
Search Tags:#mlops#big data#distributed computing#ray#spark#domino#dask
One-click install
npx skills add https://github.com/dominodatalab/domino-claude-plugin --skill domino-distributed-computing-dominodatalab
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: domino-distributed-computing
Source: https://github.com/dominodatalab/domino-claude-plugin/tree/main/skills/distributed-computing
Command: npx skills add https://github.com/dominodatalab/domino-claude-plugin --skill domino-distributed-computing-dominodatalab

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill simplifies the management and utilization of distributed computing frameworks like Apache Spark, Ray, and Dask within the Domino Data Lab environment, enabling users to efficiently process large datasets and scale complex computations.

Core Features & Use Cases

  • Framework Selection: Guidance on choosing between Spark, Ray, and Dask based on workload requirements (data processing, ML training, parallel Python).
  • Cluster Management: Instructions for launching on-demand clusters via the Domino UI and Python SDK.
  • Code Examples: Practical Python snippets for connecting to, processing data with, and training models using Spark, Ray, and Dask.
  • GPU Acceleration: How to leverage GPUs with Spark (RAPIDS) and Ray.
  • Autoscaling: Configuration and monitoring of dynamic cluster scaling.
  • Use Case: You have a multi-terabyte dataset and need to perform complex ETL operations. This Skill will guide you to launch a Spark cluster, write PySpark code to process the data efficiently, and save the results.

Quick Start

Use the domino-distributed-computing skill to launch a Spark cluster with 4 workers and process data from '/mnt/data/large_dataset/'.

Frequently Asked Questions about domino-distributed-computing

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run Spark, Ray, or Dask for distributed computing in Domino?▼

To run distributed computing in Domino, you can launch on-demand Spark, Ray, or Dask clusters directly via the Domino UI or Python SDK. The environment provides configuration guidance and Python code examples for large-scale data processing and model training.

When should I choose Ray over Spark or Dask for my workload?▼

Choose Ray for parallel Python workloads and machine learning training, Spark for large-scale data processing and ETL operations, and Dask for scaling Python analytics. The selection depends on whether your task prioritizes ML training, data manipulation, or parallel compute scaling.

Can I use GPU acceleration with distributed Spark and Ray clusters?▼

Yes, you can leverage GPU acceleration with Spark using RAPIDS and with Ray for machine learning workloads. The distributed computing setup provides guidance on configuring GPU resources for accelerated data processing and model training.

How do I configure autoscaling for distributed computing clusters?▼

You configure autoscaling for distributed computing clusters by using the dynamic cluster scaling settings available in Domino. This allows your Spark, Ray, or Dask clusters to automatically adjust compute resources based on workload demands.

What is the best way to process multi-terabyte datasets in Domino?▼

The best way to process multi-terabyte datasets is to launch a Spark cluster using the Domino UI or SDK, then write PySpark code to efficiently execute ETL operations and save the processed results.

Does launching a distributed computing cluster require manual environment setup?▼

No, launching a distributed computing cluster does not require manual environment setup. You can launch on-demand clusters with specific worker counts directly through the Domino UI or Python SDK to immediately start processing data.

Related Skills

Explore More Data & Analytics Skills→
ToXMonToXMon

domino-distributed-computing

Launch on-demand Spark, Ray, or Dask clusters in Domino.

Community
Advanced
K-Dense-AIK-Dense-AI
33.0k

modal

Run Python cloud functions with autoscaling GPU compute and web endpoints.

Official
Advanced
distribucionesfayosmoreno-sysdistribucionesfayosmoreno-sys

ml-pipeline-workflow

Orchestrate end-to-end MLOps pipelines from data preparation to deployment.

Community
Advanced
Monjyu1101Monjyu1101
1

simpo-training

Run SimPO training scripts to align LLMs without a reference model.

Community
Advanced