swe-bench-grafema-experiments

A/B testing for AI code understanding in SWE-bench requires configuring Docker containers, pre-building code graphs, and running baseline versus experimental agent conditions. This runbook provides detailed setup steps, execution guidance, and evaluation metrics for comparing agent performance with and without code graph context.

Providing an AI agent with a code graph representation can improve bug fixing and feature implementation in software engineering benchmarks by offering structured project context. This Skill enables quantitative A/B measurement of that improvement against a baseline within the SWE-bench environment.

Docker configuration for SWE-bench experiments involves setting up containers that support both baseline and experimental conditions for code analysis tools. The runbook details building dependencies, pre-building code graphs, and addresses common Docker integration pitfalls specific to Node.js environments.

Common Docker and Grafema integration pitfalls for Node.js environments include dependency conflicts and graph pre-building errors during experimental setup. The runbook addresses these specific issues alongside container configuration to ensure stable A/B test execution within the SWE-bench benchmark.

Evaluating AI agent trajectories after SWE-bench experiments involves analyzing the execution results of both baseline and experimental conditions. The runbook covers result evaluation and trajectory analysis to determine the quantitative impact of code graph context on agent performance.