ArXiv Agentic Verifier

Generating edge-case tests for competitive-coding solutions involves using an LLM to analyze problem statements and candidate code, producing targeted inputs and expected outputs to expose logic flaws. The Skill then executes the code in a sandbox environment to verify output correctness against expected results.

Verifying code correctness for Python and JavaScript algorithms is best done by generating discriminative test cases aimed at breaking incorrect logic, executing the candidate program, and enforcing pass/fail by output matching. This Skill automates that workflow using LLM-driven test generation and sandbox execution.

Yes, an OpenAI API key is required for LLM-driven test generation and code verification. The Skill depends on the OpenAI API to reason about problem statements and generate targeted edge-case inputs required for verifying competitive-coding solutions.

To automatically find a counterexample for a wrong-answer submission, this Skill analyzes the candidate code to identify likely failure modes and generates specific inputs plus expected outputs aimed at breaking incorrect logic. It then executes the code with these generated inputs to detect correctness failures.

No, this approach does not use random sampling to stress-test small algorithm implementations. Instead, it uses an LLM to reason about the problem statement and candidate code, generating targeted, discriminative test cases aimed at specific boundary conditions and logic flaws.