lsf_job_slow

Diagnose LSF/OpenLava/Volclava RUN-state jobs with no output or slow progress.

66|20|Updated Jul 7, 2020
One-click install
npx skills add https://github.com/liyanqing1987/lsfMonitor --skill lsf-job-slow
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: lsf_job_slow
Source: https://github.com/liyanqing1987/lsfMonitor/tree/main/monitor/conf/skills/lsf_job_slow
Command: npx skills add https://github.com/liyanqing1987/lsfMonitor --skill lsf-job-slow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you diagnose why an LSF/OpenLava/Volclava job stays in RUN状态 but shows no output, stalls, or progresses far slower than expected, so you can distinguish application blocking from node, system, or network bottlenecks.

Core Features & Use Cases

  • Collect job and process signals: Uses LSF commands to pull execution host, runtime, CPU time, memory limits/usage, working directory, and the latest stdout/stderr evidence.
  • Diagnose execution-node pressure and IO bottlenecks: Checks node load (CPU/memory/I-O/tmp space indicators) to identify overload, memory exhaustion/swap, sustained IO wait, or disk-full symptoms.
  • Classify process state and likely root cause: Interprets Linux process stat (especially D-state for uninterruptible IO sleep) to pinpoint NFS/network storage hangs or disk IO failures.
  • Summarize evidence into actionable recommendations: Produces a structured report with likely root causes, immediate mitigation, and prevention guidance.

Use Case: A batch job appears RUN for hours with no new stdout/stderr, and you need to quickly determine whether it is blocked on NFS, starved by node overload, waiting for license, deadlocked, or simply slow.

Quick Start

Ask the AI to diagnose the stuck job by telling it the job id you want to investigate.

Frequently Asked Questions about lsf_job_slow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
Why does my LSF job stay in RUN state but produce no output?

An LSF job staying in RUN state with no output often indicates a process blocked on uninterruptible I/O, such as an NFS hang, or waiting on a license. Diagnose this by checking process states for D-state and reviewing node resource pressure.

How do I diagnose an LSF job that is running abnormally slow?

To diagnose a slow LSF job, collect job signals using bjobs and bpeek, inspect execution node load with lsload and bhosts, and check process states via ps. This identifies whether node resource contention, memory exhaustion, or sustained I/O wait is causing the bottleneck.

How do I check if an HPC job is stuck on uninterruptible I/O or an NFS hang?

Check if an HPC job is stuck on uninterruptible I/O by inspecting Linux process stat output for the D-state. This state indicates the process is blocked on I/O, commonly caused by NFS or network storage hangs and disk failures.

What is the best way to identify node resource contention affecting batch jobs?

Identify node resource contention by inspecting execution node load indicators like CPU, memory, and I/O space using lsload and bhosts. This reveals whether overload, memory swap, or disk-full symptoms are starving your batch jobs.

Can I use LSF commands to distinguish between a deadlocked application and a network storage issue?

Yes, you can use LSF commands like bjobs and bpeek alongside process state checks to distinguish a deadlock from a network storage issue. Interpreting Linux process states reveals whether the stall is application-bound or caused by NFS storage hangs.