rca-annotator

What problem does it solve?

This skill helps human reviewers validate and label automated root-cause-analysis outputs by capturing whether the agent diagnosis is correct, logging precise evidence traceability, scoring difficulty, and recording alternative hypotheses for dataset building and evaluation.

Core Features & Use Cases

• Interactive annotation workflow: Presents the agent's final diagnosis and walks a user through category confirmation, summary accuracy, evidence validation, difficulty calibration, and alternative diagnosis capture.
• Evidence traceability: Requires and enforces source_file, json_path, exact_value/quote and line references so each evidence item is auditable.
• Jumpbox synchronization: Optional download and upload of analysis artifacts and annotations via SSH/rsync/scp for environments that use a remote MLflow artifact store.
• Use case: Manually curate ground-truth labels for RCA agent outputs to build benchmarks, iterate agent improvements, or produce training data.

Quick Start

Run an interactive annotation session for job 1234567 to review the agent's step5_summary, label evidence and difficulty, and save annotation.json to the .analysis/1234567 directory.