run-benchmark

You can run a LongMemEval benchmark by executing the benchmark command, which automates hypothesis generation and scoring using your specified target and judge models to streamline model evaluation.

LongMemEval benchmark scoring evaluates models by automating hypothesis generation against a dataset, tracking completed items, and generating scorecards to report the final benchmark results.

Yes, the benchmark execution supports checkpointing and resumable runs, allowing you to resume progress by marking completed dataset items, with a default limit of 500 items per run.

The benchmark tool handles dataset filtering and item tracking automatically, marking completed items during the evaluation run to manage data and monitor progress effectively.

Yes, flexible model configuration options allow you to select specific target and judge models, including choosing model variants and specifics to customize the benchmark evaluation.

The default limit for a LongMemEval benchmark run is 500 items, which helps manage dataset processing and checkpointing during model evaluation and scoring.