audit-tool-usage-from-sessions

Audit agent session logs to diagnose why bash is preferred over native tools.

10|1|Updated Jul 7, 2026
One-click install
npx skills add https://github.com/catalystctl/catcode --skill audit-tool-usage-from-sessions-catalystctl
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: audit-tool-usage-from-sessions
Source: https://github.com/catalystctl/catcode/tree/main/.catalyst-code/skills/audit-tool-usage-from-sessions
Command: npx skills add https://github.com/catalystctl/catcode --skill audit-tool-usage-from-sessions-catalystctl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Coding agents often invoke grep, find, or awk through bash instead of using the native tool equivalents, and it is hard to tell whether the cause is a missing capability or simple habit. This Skill audits all recorded agent sessions to classify every bash-based search call, so you can decide whether to fix tool code, fix the schema description, or leave behavior alone. ## Core Features & Use Cases - Session-wide tool usage audit: Parses every session JSONL log, extracting tool_calls and content-block tool_use shapes to count native tool calls versus bash invocations of grep, rg, find, awk, and sed. - Cause classification with tags: Labels each bash search call with tags like OUTSIDE_WS, PIPE, FLAG_V, FLAG_F, FIND, and PLAIN_INWS to separate real capability gaps from habit and legitimate workspace-confinement cases. - Gap remediation workflow: Guides implementing missing flags in both the rg-backed and pure-Rust code paths, adding unit tests, running cargo test and clippy, and enriching the schema description to surface existing capabilities. - Use Case: You notice your agent keeps running grep through bash. Run this audit across 661 sessions and discover that invert (-v) and word (-w) flags are real gaps to implement, while pipe-to-head usage is an awareness problem fixed by improving the tool's schema description. ## Quick Start Audit all my agent sessions and tell me why the agent uses grep via bash instead of the native grep tool.

Frequently Asked Questions about audit-tool-usage-from-sessions

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find out why my agent uses bash instead of native tools?

Audit the session JSONL logs under ~/.config/catalyst-code/sessions and parse each assistant message's tool_calls function name and arguments. Classify bash calls invoking grep, rg, find, awk, or sed by cause tags to see whether the driver is a missing flag, a pipe habit, or out-of-workspace paths.

How to audit agent session logs for tool usage patterns?

Write a Python script that walks every session .jsonl file, parses tool_calls[].function.arguments as a JSON string, and also handles older content-block tool_use shapes. Count native tool calls for proportion and tag each bash search call, keeping examples per tag.

Should I add missing grep flags to the native tool or fix the description?

Tags absent from the native tool, like invert (-v) or fixed-string (-F), are real gaps requiring code changes in both the rg and pure-Rust paths. Tags the tool already supports, like head_limit pagination replacing pipe-to-head, are awareness problems fixed by enriching the schema description.

Why does the agent use bash grep for paths outside the workspace?

Out-of-workspace access via bash is deliberate workspace confinement, not a bug. Native tools are path-confined by design, so OUTSIDE_WS cases like system, proc, or sibling repository searches should be left as-is rather than weakening confinement.

What are the limitations of session-based tool usage audits?

The audit only sees recorded sessions and must exclude the live session you are running in. Schema definitions can also diverge from implementation, so you must read both the schema definitions and the tool dispatch code before concluding a capability is missing.