skills.rest
ruvnet
ruvnetCOMMUNITY
Explore all skills by this author →
Website𝕏@ruvGitHub

cost-counterfactual

Compares actual session spend against hypothetical always-haiku, always-sonnet, and always-opus routing baselines.

70.1k|8.4k|Updated Jun 2, 2025
Search Tags:#savings#counterfactual#LLM routing#token pricing#Observability#Cost Analysis#model baselines
One-click install
npx skills add https://github.com/ruvnet/claude-flow --skill cost-counterfactual
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: cost-counterfactual
Source: https://github.com/ruvnet/claude-flow/tree/main/plugins/ruflo-cost-tracker/skills/cost-counterfactual
Command: npx skills add https://github.com/ruvnet/claude-flow --skill cost-counterfactual

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Teams using model routing cannot easily tell whether the router is actually saving money or over-escalating tasks to expensive models. This Skill quantifies the router's value by comparing real session spend against what the same tokens would have cost under a single-model strategy.

Core Features & Use Cases

  • Multi-Baseline Comparison: Computes hypothetical spend for always-haiku, always-sonnet, and always-opus baselines using per-tier token pricing.
  • Savings Quantification: Reports per-baseline savings and percentages, where negative always-haiku savings flag over-escalation and positive always-sonnet savings prove routing value.
  • Use Case: In a quarterly cost review, run the analysis over the last 90 days to state "we saved $X versus an always-Sonnet strategy," or wire it into CI to fail builds when routing savings drop below 30%.

Quick Start

Ask the assistant to run a counterfactual cost analysis over the last 7 days comparing actual spend against all single-model baselines.

Frequently Asked Questions about cost-counterfactual

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I measure whether LLM model routing is saving money?▼

Run a counterfactual cost analysis that reprices your actual session tokens under single-model baselines like always-sonnet. The difference between hypothetical and actual spend quantifies the router's savings as a dollar amount and percentage.

What does negative savings against an always-haiku baseline mean?▼

Negative always-haiku savings means the router chose more expensive models than haiku for tasks haiku could have handled, signaling over-escalation. Investigate by lowering the qualityBar or inspecting individual sessions to see if the escalation was warranted.

Can I use counterfactual cost analysis as a CI gate?▼

Yes, output the analysis with --format json and pipe it to jq to assert a savings threshold, such as failing the build when savings versus the sonnet baseline drop below 30 percent. This acts as a workload-shift detector.

What are the limitations of counterfactual cost analysis?▼

The analysis assumes the same tokens at the same complexity would produce the same outcome from the baseline model, which is an upper bound. It does not capture baseline failures or retries, so treat the numbers as a quality-blind ceiling.

How do I filter the cost analysis to a specific time window?▼

Pass the --since flag with a duration such as 7d to restrict the analysis to recent sessions. Without the flag, the analysis covers all recorded sessions in the cost-tracking namespace.

Related Skills

Explore More Data & Analytics Skills→
zeuiklizeuikli
156

cost-tracker

Calculate session token usage and convert it to USD cost estimates.

Community
Intermediate
maziyarpanahimaziyarpanahi
5.0k

gating-deid-leakage

Fail CI when PHI recall drops below approved thresholds or critical identifiers leak.

Community
Advanced
mahg-esmahg-es
3

budget-forecasting

Forecast cloud and infrastructure budgets from historical costs using linear regression.

Official
Intermediate
ignidusignidus

model-routing

Routes tasks to cheapest of Cupertino's Haiku, Sonnet, Opus meeting quality needs.

Official
Advanced