skills.rest
yale-som-hpc
yale-som-hpcOFFICIAL
Explore all skills by this author →
GitHub

accelerating-python

Profile Python Slurm jobs to identify bottlenecks and guide acceleration.

5|1|Updated Apr 29, 2026
Search Tags:#python#profiling#duckdb#polars#hpc#slurm#numba
One-click install
npx skills add https://github.com/yale-som-hpc/claude-code-marketplace --skill accelerating-python
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: accelerating-python
Source: https://github.com/yale-som-hpc/claude-code-marketplace/tree/main/plugins/hpc/skills/accelerating-python
Command: npx skills add https://github.com/yale-som-hpc/claude-code-marketplace --skill accelerating-python

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Profile-first acceleration for Python on Yale's HPC cluster by identifying bottlenecks and guiding targeted optimizations before resorting to multiprocessing or GPUs.

Core Features & Use Cases

  • Profiling-driven bottleneck identification in Python Slurm jobs.
  • Evaluation of acceleration options (DuckDB/Polars/Numba) and disciplined use of parallelism.
  • Real-world scenario: accelerate a data-processing workflow that reads Parquet data and performs complex transformations on the cluster.

Quick Start

Run a profiler on a Python Slurm job and apply the recommended acceleration steps.

Frequently Asked Questions about accelerating-python

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I profile Python Slurm jobs to identify bottlenecks?▼

Profile Python Slurm jobs using tools like cProfile or py-spy to pinpoint bottlenecks. This process reveals exact CPU-bound or memory-heavy constraints, guiding targeted acceleration decisions for your cluster workloads.

What is the best way to accelerate Python data processing on an HPC cluster?▼

The best way to accelerate Python on HPC is a profile-first approach: evaluate DuckDB or Polars for data transformations, apply Numba for CPU-bound tasks, and use disciplined parallelism before resorting to GPUs.

When should I use DuckDB or Polars instead of multiprocessing in Python?▼

Use DuckDB or Polars instead of multiprocessing when profiling indicates memory-heavy bottlenecks reading Parquet data. These engines process complex transformations efficiently, often outperforming multiprocessing before GPUs are considered.

Can I use Numba to optimize CPU-bound Python workloads on Yale SOM HPC?▼

Yes, you can use Numba to optimize CPU-bound Python workloads on Yale SOM HPC. Profiling must first confirm the bottleneck is CPU-bound, ensuring Numba's JIT compilation provides targeted acceleration without unnecessary complexity.

Do I need to profile my Python code before applying GPU acceleration?▼

Yes, you must profile Python code before applying GPU acceleration. Profiling establishes guardrails to determine if bottlenecks justify GPU resources, ensuring disciplined parallelism and engine choices are exhausted first.

Related Skills

Explore More Software Engineering Skills→
gaebalaigaebalai
2

raspi-can-bootstrap

Automate CAN bootstrapping for Raspberry Pi vehicle diagnostics environments.

Community
Advanced
ovachieverovachiever
52

ray-data

Process distributed ML data pipelines with Ray Data across CPU/GPU clusters.

Community
Advanced
tronghieutronghieu
36

python-performance-optimization

Profile Python code with cProfile and py-spy to optimize CPU and memory usage.

Community
Intermediate
ovachieverovachiever
52

pytorch-fsdp

Apply PyTorch FSDP for sharded, mixed-precision distributed training of large models.

Community
Advanced