Home Security AI Benchmark

To benchmark LLM and VLM models for home security AI, run the suite via `node scripts/run-benchmark.cjs --vlm http://localhost:5405` to evaluate scene analysis, threat detection, and alert triage. It tests 143 scenarios across 16 categories to measure model accuracy and reliability.

Prompt injection resistance testing evaluates whether an AI model can withstand malicious inputs designed to alter its behavior. This benchmark includes specific tests to ensure LLMs maintain security classification and alert routing integrity when subjected to adversarial prompt injection attacks.

Yes, you can evaluate vision-language models for scene analysis by running the benchmark with a VLM server address. The suite assesses visual understanding capabilities, including VLM-to-alert triage and scene analysis, generating detailed HTML reports with pass/fail scores and latency charts.

Yes, the benchmark requires the OpenAI dependency to execute its comprehensive test suite. It uses this dependency to evaluate LLM and VLM capabilities, generating detailed reports on token usage, latency charts, and pass/fail scores for security classification and context preprocessing.

The best way to compare AI models for security camera alert triage is using a comprehensive benchmark suite that generates an HTML report with pass/fail scores, latency charts, and token usage. This allows security system integrators to determine optimal model accuracy and reliability before deployment.

You can integrate benchmark results with Aegis by setting environment variables for automated setup and reporting. This allows the benchmark to automatically evaluate context preprocessing, tool use, and security classification, routing the results directly into your Aegis system.