skills.rest
kreuzberg-dev
kreuzberg-devOFFICIAL
Explore all skills by this author →
WebsiteGitHub

benchmark-workflow

Run and diagnose Xberg extraction benchmarks, quality scoring, and ground-truth fixtures.

9.2k|581|Updated Jan 31, 2025
Search Tags:#benchmarking#diagnosis#ground truth#extraction#Fixtures#quality scoring#GitHub workflow
One-click install
npx skills add https://github.com/kreuzberg-dev/kreuzberg --skill benchmark-workflow
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: benchmark-workflow
Source: https://github.com/kreuzberg-dev/kreuzberg/tree/main/.ai-rulez/skills/benchmark-workflow
Command: npx skills add https://github.com/kreuzberg-dev/kreuzberg --skill benchmark-workflow

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Benchmarking a document extraction library requires trustworthy ground truth, reproducible runs, and careful diagnosis of failures. This Skill guides you through running, diagnosing, and changing the Xberg benchmark harness without corrupting ground truth or misreading results.

Core Features & Use Cases

  • Ground-Truth Integrity: Enforces independently sourced ground truth (manual, vision, pdf_text_layer, pandoc, python-docx) recorded in each fixture's ground_truth.source field, never Xberg's own extractor output.
  • Run Diagnosis: Separates infrastructure failures (missing backends, absent fixtures, malformed artifacts) from genuine extraction or quality regressions, and inspects per-adapter artifacts before aggregate jobs.
  • Fair A/B Comparison: Requires identical corpus, config, renderer, cache state, and metric on control and experiment, with extraction and OCR caches invalidated before behavior-changing runs.
  • Use Case: Before dispatching the dispatch-only .github/workflows/benchmarks.yaml workflow, validate fixtures with the harness validate-gt command and add focused tests for any report or artifact contract changes.

Quick Start

Diagnose why the latest benchmark run shows a quality regression for the PDF adapter before dispatching a new workflow run.

Frequently Asked Questions about benchmark-workflow

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run the Xberg extraction benchmarks?▼

The benchmark system lives in tools/benchmark-harness and runs through the dispatch-only .github/workflows/benchmarks.yaml workflow. It does not run on push or gate merges, so trigger it manually and treat results as evidence for that exact commit SHA and inputs.

How do I create ground truth for extraction benchmarks?▼

Never use Xberg's own extractor output as ground truth. Use an independent source such as manual annotation, vision, pdf_text_layer, pandoc, or python-docx, record it in the fixture's ground_truth.source field, and validate with the harness validate-gt command.

Why did my benchmark run fail with an artifact contract error?▼

Aggregate contract failures often come from missing or unexpectedly named per-adapter artifacts even when individual adapters ran. Inspect the per-adapter artifacts before the aggregate job, and separate infrastructure failures from actual extraction quality problems.

How do I run a fair A/B benchmark comparison?▼

A quality claim requires the same corpus, config, renderer, cache state, and metric on both control and experiment. Disable or invalidate extraction and OCR caches before A/B runs whose output behavior changed.

Why do OCR benchmark word counts look wrong?▼

Compare accepted OCR pages before raw word counts, because rejected OCR pages contribute neither text nor structured paragraphs. Also measure headings and lists using Markdown output, since plain output normalizes away list markers.

Related Skills

Explore More Software Engineering Skills→
Growth4U-systemsGrowth4U-systems

docx

Create, edit, and analyze .docx files using docx-js, pandoc, and LibreOffice.

Official
Advanced
webrix-aiwebrix-ai
1

csat-tracking

Track CSAT scores across support channels and flag below-threshold trends.

Official
Intermediate
mbwsimsmbwsims
71

test

Generate unit and integration tests with deep assertions across JavaScript, Python, and Go.

Community
Advanced
GYF0311GYF0311
5

skillify

Convert ad-hoc features into reusable agent skills with SKILL.md contracts.

Community
Advanced