skills.rest
Arm-Examples
Arm-ExamplesOFFICIAL
Explore all skills by this author →
GitHub

llm-debug-test-failures

Debug failing LLM integration tests with verbose ctest reruns and Python scripts.

4|3|Updated Feb 18, 2026
Search Tags:#testing#debugging#integration tests#llm#model output#backend regressions
One-click install
npx skills add https://github.com/Arm-Examples/LLM-Runner --skill llm-debug-test-failures
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llm-debug-test-failures
Source: https://github.com/Arm-Examples/LLM-Runner/tree/main/skills/llm-debug-test-failures
Command: npx skills add https://github.com/Arm-Examples/LLM-Runner --skill llm-debug-test-failures

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill helps developers efficiently diagnose and resolve failing integration tests for Large Language Models (LLMs), pinpointing issues related to model output, configuration, or backend regressions.

Core Features & Use Cases

  • Reproduce Failing Tests: Easily re-run specific failing tests with verbose output.
  • Inspect Model Responses: Capture detailed logs of prompts, responses, and runtime parameters for analysis.
  • Validate Configurations: Verify model configuration files, paths, and runtime settings like context size and batch size.
  • Trace Issues: Step through backend integrations (llama.cpp, ONNX Runtime GenAI, MediaPipe, MNN) and upstream framework sources to identify bugs.
  • Use Case: When an llm-cpp-ctest fails due to unexpected model output, use this Skill to rerun the test, capture the exact prompt and response, and inspect the configuration to understand why the output drifted.

Quick Start

Rerun the failing LLM integration tests verbosely from your build directory.

Frequently Asked Questions about llm-debug-test-failures

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug failing LLM integration tests caused by unexpected model output drift?▼

Debug failing LLM integration tests by re-running specific tests verbosely to capture detailed logs of prompts, responses, and runtime parameters. This isolates whether model output drift stems from configuration errors or backend regressions.

How do I inspect LLM model responses and runtime parameters during integration testing?▼

Inspect LLM model responses and runtime parameters by utilizing Python scripts to capture detailed logs of prompts and outputs during test execution. This helps trace issues into backend framework code like llama.cpp or ONNX Runtime GenAI.

Why does my ctest fail due to prompt or template mismatches in the LLM configuration?▼

Ctest failures from prompt or template mismatches occur when model configuration files, paths, and runtime settings like context size are invalid. Validate these configurations and re-run tests verbosely to pinpoint the exact mismatch.

Can I trace backend regressions in frameworks like llama.cpp or ONNX Runtime GenAI?▼

You can trace backend regressions in frameworks like llama.cpp, ONNX Runtime GenAI, MediaPipe, and MNN by stepping through backend integrations and upstream framework sources. This identifies bugs causing integration test failures.

What is the best way to reproduce a failing llm-cpp-ctest with verbose output?▼

The best way to reproduce a failing llm-cpp-ctest is to re-run the specific failing test verbosely from your build directory. This captures the exact prompt, response, and configuration to understand why the output drifted.

Related Skills

Explore More Software Engineering Skills→
steven715steven715
2

tdd-workflow

Enforce test-driven development in C++ projects using Google Test and CMake/CTest.

Community
Intermediate
nathanielcrowell12-specnathanielcrowell12-spec

testing

Automate Playwright and Vitest tests for PhotoVault user journeys.

Community
Intermediate
incidentfoxincidentfox
654

observability

Correlate logs, metrics, and traces across multiple backends to identify root causes.

Official
Intermediate
awslabsawslabs
15

run-source-model-baseline

Generate live source-model baselines from golden prompt datasets for LLM migration evaluation.

Official
Advanced