ark-chainsaw-testing

Run end-to-end tests for Ark resources using Chainsaw by executing test suites organized under the tests/ directory with multiple scenarios, including mock-llm configurations and manifests.

The standard structure for organizing Chainsaw tests involves placing test scenarios within a tests/ directory and maintaining associated configurations and resources within a manifests/ directory.

Mock-llm configurations help debug Chainsaw test failures by providing deterministic LLM responses, ensuring reusable and predictable test patterns for verifying end-to-end behavior.

Yes, you can write deterministic e2e tests for Ark resources by enforcing standard test structures and applying mock-llm configurations to ensure consistent and repeatable test execution.

Chainsaw e2e tests fail with non-deterministic LLM responses because they lack a mock-llm configuration, which is required to enforce deterministic patterns and reusable test behaviors.

Yes, this testing approach supports multiple scenarios within a single suite, specifically targeting test suites organized under tests/ with various mock-llm configurations and manifests.