arabic-agent-eval

The benchmark evaluates six categories: simple call, parameter extraction, multi-step reasoning, dialect handling, tool choice, and error handling, measuring how accurately language models invoke tools across various Arabic dialects and providers.

Yes, the quick evaluation mode runs a fast, single-provider check to gauge tool invocation behavior for language models using the aae quick openai command in your Python environment.

You need a Python environment with the aae CLI installed to execute the Arabic tool-calling benchmark and evaluate language model performance across dialects and providers.

The benchmark includes a dedicated dialect handling category within its full evaluation mode, testing how accurately language models invoke tools across different Arabic dialect variations and providers.

Quick evaluation runs a fast single-provider check, while full benchmark mode executes a comprehensive run across multiple providers and dialects to compare aggregated performance metrics across six categories.