compute-kernel

Connect Jupyter kernels running on HPC compute nodes via SLURM job submission.

1|Updated Oct 25, 2020
One-click install
npx skills add https://github.com/bfairkun/dotfiles --skill compute-kernel-bfairkun
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: compute-kernel
Source: https://github.com/bfairkun/dotfiles/tree/main/agents/.agents/skills/compute-kernel
Command: npx skills add https://github.com/bfairkun/dotfiles --skill compute-kernel-bfairkun

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Login-node Jupyter kernels on shared HPC systems like RCC Midway are silently OOM-killed because the memory ceiling applies to the whole user across all sessions, causing lost variables and reset cell counters. This Skill moves kernels onto compute nodes with dedicated memory and provides recovery workflows when kernels die. ## Core Features & Use Cases - Compute-node kernel startup: Submit SLURM sbatch jobs that launch ipykernel bound to the node's routable IP, with partition and account handling for RCC Midway and UMich Great Lakes. - Kernel lifecycle management: Detect stale kernels, kill old SLURM jobs, connect via connection files, and verify hostname and Python executable. - Wall-time autosave and recovery: Automatically snapshot notebook state every 30 minutes and restore sessions into a fresh kernel after a job ends or a kernel dies. - Use Case: A pandas kernel on the Midway login node is OOM-killed mid-analysis. Use this Skill to submit a 48G caslake compute job, connect to its kernel, restore the autosaved session, and continue the notebook without re-running expensive steps. ## Quick Start Connect to a compute kernel on Midway with 48G of memory so my notebook stops getting OOM-killed on the login node.

Frequently Asked Questions about compute-kernel

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run a Jupyter kernel on an HPC compute node?▼

Submit a SLURM sbatch job that activates your conda environment and launches ipykernel with --ip set to the node's routable IP, writing a connection file to scratch. Then connect to that connection file from the login node and verify with socket.gethostname().

Why does my Jupyter kernel keep dying on the login node?▼

On RCC Midway the login node caps the entire user at 8 GiB via a cgroup shared across all sessions, so accumulated usage triggers the OOM killer silently. Check memory.max_usage_in_bytes in your cgroup slice; if it equals the limit, move to a compute-node kernel instead of retrying.

Why can't I connect to my SLURM kernel job that is running?▼

ipykernel defaults to binding 127.0.0.1, which is unreachable from the login node where the MCP server runs. Check the connection file's ip field; if it is loopback, resubmit the job binding the node's routable IP obtained from ip route get 8.8.8.8.

How do I recover a dead Jupyter kernel session?▼

List autosave .pkl files in your scratch directory, start a new compute kernel, then call walltime_autosave.load with the notebook name to restore variables. Restart the autosave monitor afterward so the new job keeps snapshotting state.

Can I run an R kernel on a compute node the same way?▼

Yes, the same workflow applies using the agent_r_kernel.json connection file variant. Once connected, send plain R code through run_python since the kernel itself is R and no %%R magic is needed.

How do I size memory for rerunning an existing notebook?▼

Read the render-env comment block in the notebook's last rendered HTML, which records peak_rss, node_type, partition, and allocated resources. Use peak_rss plus headroom to choose the --mem value for the new kernel job.