skills.rest
surus-lat
surus-latOFFICIAL
Explore all skills by this author →
Website𝕏@surusdevGitHub

transcription-benchmark

Benchmark local ASR models on FLEURS datasets with WER and CER tables.

8|Updated Sep 12, 2025
Search Tags:#benchmarking#evaluation#transcription#whisper#speech-to-text#asr
One-click install
npx skills add https://github.com/surus-lat/benchy --skill transcription-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: transcription-benchmark
Source: https://github.com/surus-lat/benchy/tree/main/.agent/skills/transcription-benchmark
Command: npx skills add https://github.com/surus-lat/benchy --skill transcription-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, nemo-toolkit, torch, and includes scripts (resource) components.

What problem does it solve?

This skill solves the difficulty of comparing transcription model performance across different architectures by providing a standardized, local-only benchmarking suite that eliminates cloud dependency and ensures reproducible results.

Core Features & Use Cases

  • Multi-Architecture Comparison: Evaluate Whisper, Surus LATAM, Canary, Qwen3-ASR, and Voxtral models on a unified FLEURS dataset.
  • Local Execution: Runs entirely on your hardware, ensuring data privacy and offline reproducibility without needing paid API endpoints.
  • Automated Reporting: Generates comparative WER and CER tables to identify the best-performing model for specific languages like Spanish and Portuguese.

Quick Start

Run the transcription benchmark to evaluate the full panel of supported models on the FLEURS dataset.

Frequently Asked Questions about transcription-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark local speech-to-text models for Spanish and Portuguese?▼

You can benchmark local speech-to-text models by evaluating Whisper, Canary, Qwen3-ASR, and Voxtral on FLEURS datasets to generate comparative WER and CER tables for languages like Spanish and Portuguese.

Does local ASR benchmarking require cloud API access or paid endpoints?▼

Local ASR benchmarking does not require cloud API access or paid endpoints. It runs entirely on your local hardware, ensuring data privacy and offline reproducibility using HuggingFace model weights.

What hardware do I need to run automated speech recognition evaluation locally?▼

Running automated speech recognition evaluation locally requires specific local GPU hardware to process the standardized transcription tasks across multiple architectures like Whisper and Voxtral.

Can I compare Whisper performance against Canary and Voxtral on a unified dataset?▼

Yes, you can compare Whisper performance against Canary, Qwen3-ASR, and Voxtral on a unified FLEURS dataset, which provides a standardized environment for comparative analysis.

What metrics are generated when evaluating transcription model architectures?▼

Evaluating transcription model architectures generates comparative WER and CER tables, allowing you to identify the best-performing model for specific languages within the evaluated dataset.

Related Skills

Explore More Software Engineering Skills→
surus-latsurus-lat
8

whisper-benchmark

Benchmark Whisper-family ASR models on FLEURS datasets with WER and CER metrics.

Official
Advanced
sayalinvidiasayalinvidia

digital-health-clinical-asr-eval

Score NeMo-format clinical ASR manifests and generate a five-section KER leaderboard.

Community
Advanced
zangjeicyzangjeicy

whisper

Transcribe multilingual audio to text using OpenAI's Whisper model.

Community
Intermediate
ciniencecinience
396

alicloud-ai-audio-asr

Transcribe non-realtime speech from audio files using Alibaba Cloud Qwen ASR models.

Community
Intermediate