skills.rest
bfairkun
bfairkunCOMMUNITY
Explore all skills by this author →
GitHub

parse-logs

Diagnose Snakemake and Slurm pipeline failures by parsing HPC job logs.

1|Updated Oct 25, 2020
Search Tags:#SLURM#HPC#error diagnosis#pipeline debugging#Snakemake#log parsing#exit codes
One-click install
npx skills add https://github.com/bfairkun/dotfiles --skill parse-logs-bfairkun
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: parse-logs
Source: https://github.com/bfairkun/dotfiles/tree/main/agents/.agents/skills/parse-logs
Command: npx skills add https://github.com/bfairkun/dotfiles --skill parse-logs-bfairkun

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? When a Snakemake pipeline or Slurm job fails on an HPC cluster, the error is buried across multiple log files, and finding the root cause manually is slow and error-prone. ## Core Features & Use Cases - Failure Localization: Find the failing rule in Snakemake run logs and read the corresponding per-rule log file. - Slurm Job Inspection: Check job state, exit codes, and resource usage with sacct and scontrol to identify OOM or timeout failures. - Pattern-Based Diagnosis: Match common failure patterns such as out-of-memory, timeout, MissingInputException, conda env build failures, and MissingOutputException. - Use Case: A Snakemake run on RCC Midway errors out overnight; use this Skill to locate the failing rule, read its log, check the Slurm exit code, and determine whether to increase memory or fix a missing input. ## Quick Start Ask the assistant to diagnose why Snakemake job 12345 failed and explain the error from the logs.

Frequently Asked Questions about parse-logs

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find which Snakemake rule failed?▼

Search the Snakemake run log in .snakemake/log/ for lines containing Error or rule using grep. The output names the failing rule and points to its per-rule log file under logs/, which contains the actual error details.

How to check why a Slurm job failed with sacct?▼

Run sacct -j <jobid> --format=JobID,State,Elapsed,MaxRSS,ExitCode,CPUTime to see the job state and exit code. Exit code 137 indicates out-of-memory, while a TIMEOUT state means the time limit was exceeded.

What does Slurm exit code 137 mean?▼

Exit code 137 means the job was killed for exceeding its memory allocation, shown as OUT_OF_MEMORY or FAILED state. The fix is to increase mem_mb in the rule's resources block and resubmit.

Why does Snakemake report MissingOutputException?▼

MissingOutputException means the job ran but did not produce the expected output file. Read the rule's log file fully to find the underlying error, since the command likely failed silently or wrote output elsewhere.

Does this log parsing approach work outside RCC Midway?▼

The Skill is scoped to RCC Midway HPC, but the techniques are standard Snakemake and Slurm commands. Log directory conventions like logs/<rulename>.log may differ on other clusters depending on pipeline configuration.

Related Skills

Explore More Software Engineering Skills→
sahunosahuno

snakemake

Automate creation and debugging of Snakemake workflows for SLURM-based HPC.

Community
Advanced
projectbluefinprojectbluefin
11

argo-workflows

Author, lint, and debug Argo Workflows and WorkflowTemplates for Kubernetes CI pipelines.

Official
Advanced
KwongFukKwongFuk

slurm-auto-retry-watchdog

Submit Slurm sbatch scripts and retry retriable failures up to a fixed limit.

Community
Advanced
marin-communitymarin-community
3.2k

scan-logs

Scan oversized logs with Gemini-backed grep and summarization modes.

Official
Intermediate