dw-skill-eval-score

Automating transcript scoring involves combining rule-based checks, LLM judgments, and statistical aggregation to produce a transparent scores.yaml file. This pipeline evaluates API call correctness, order, and parameters while applying LLM-based judging with calibration checks.

Running an LLM judge evaluation pipeline requires a Python environment with access to eval/config.yaml, testset.yaml, and transcripts. The pipeline applies rule-based checks and multiple LLM judges with redacted model information to generate robust, calibrated evaluation scores.

Yes, generating scores.yaml from transcripts requires a Python environment. The evaluation pipeline leverages built-in eval/scripts modules to perform rule-based checks, LLM judgments, and layered statistics to calculate and aggregate run scores per case and per model.

Yes, you can calibrate LLM judge scores during skill evaluation. The pipeline applies calibration checks alongside rule-based evaluations and statistical aggregation, ensuring robust and auditable quality control when summarizing run scores per case and per model.

Evaluating transcript API calls, order, and parameters is handled through rule-based checks within the evaluation pipeline. This ensures correct usage and safe operations before combining the results with LLM judgments and statistical aggregation to produce a final scores.yaml.

End-to-end three-level scoring includes rule-based evaluation of transcript API calls, LLM-based judging with redacted model information and multiple judges, and layered statistics with calibration checks. These levels summarize run scores per case and per model to generate an auditable scores.yaml.