sv-eval

To run network log anomaly detection evaluations, you execute reproducible tests for the E1 environment using specific models and example counts. This generates detailed metrics reports including Accuracy, ECE, FN%, and FP% for performance analysis.

Yes, you can compare multiple AI models on configuration verification tasks by running evaluations in the E2 environment. The comparison generates metrics reports analyzing MeanReward, FormatSuccess%, and AvgTools to identify the best performing model.

Security verifier evaluation metrics include Accuracy, ECE, FN%, and FP% for network log anomaly detection (E1), and MeanReward, FormatSuccess%, and AvgTools for configuration verification (E2). These metrics provide detailed insights into model performance and areas for improvement.

Yes, you need API keys for model providers to run security verifier evaluations. Optional Weave logging can also be configured to track and analyze the evaluation runs, but the primary requirement is having the necessary API keys for the models being tested.

Multi-model comparison for security anomaly detection works by executing evaluation runs for each model separately, then generating a consolidated metrics report. This allows you to directly compare results like Accuracy and FN% across different models on the same task.

The E1 environment focuses on network log anomaly detection, measuring Accuracy, ECE, FN%, and FP%. The E2 environment handles configuration verification, tracking MeanReward, FormatSuccess%, and AvgTools to evaluate how well models perform security configuration tasks.