hugging-face-evaluation-manager

To update Hugging Face model cards with benchmark scores, this Skill parses evaluation tables directly from README files and converts them into model-index YAML metadata. This process ensures your model cards are up-to-date and compatible with Hugging Face leaderboards.

Model-index metadata structures benchmark scores for seamless integration with Hugging Face's ecosystem. It enables leaderboards and discovery platforms to automatically parse and display your model's evaluation results.

Yes, you can import model evaluation scores from the Artificial Analysis API. The Skill manages evaluation results by extracting data from README files and importing benchmark scores directly from the Artificial Analysis API into your model metadata.

Yes, this evaluation manager supports running custom benchmarks with vLLM or lighteval. It integrates with both frameworks alongside inspect-ai and transformers to execute custom evaluations and update your model's benchmark scores.

To format MMLU and GSM8K results into YAML, the Skill parses benchmark scores from your README and generates compliant model-index YAML metadata. This allows you to automatically create a pull request to update the model card.

You need vLLM and torch installed if you plan to run custom evaluations locally. However, if you are only extracting data from READMEs or importing scores from the Artificial Analysis API, these specific dependencies are not actively required for the core parsing tasks.