skills.rest
nvtruongops
nvtruongopsCOMMUNITY
Explore all skills by this author →
GitHub

guardrail-evaluation-metrics

Computes classification metrics, adversarial robustness scores, and latency benchmarks for prompt injection guardrails.

1|Updated Sep 1, 2026
Search Tags:#prompt injection#scikit-learn#evaluation metrics#roc-auc#adversarial robustness#latency benchmark#false positive rate
One-click install
npx skills add https://github.com/nvtruongops/pi-guard --skill guardrail-evaluation-metrics-nvtruongops
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: guardrail-evaluation-metrics
Source: https://github.com/nvtruongops/pi-guard/tree/main/.agents/skills/guardrail-evaluation-metrics
Command: npx skills add https://github.com/nvtruongops/pi-guard --skill guardrail-evaluation-metrics-nvtruongops

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires scikit-learn, numpy.

What problem does it solve? Evaluating an LLM prompt-injection guardrail requires balancing attack detection against false positives on benign prompts, and ad-hoc testing cannot prove a model meets targets like FPR < 1.5% or P95 latency < 30ms. This Skill standardizes the metric calculations, adversarial robustness testing, and latency benchmarking needed to rigorously assess guardrail models. ## Core Features & Use Cases - Standard Metric Computation: Calculate accuracy, precision, recall, F1, FPR, and ROC-AUC from confusion matrices using scikit-learn. - Adversarial Robustness Suite: Benchmark models against obfuscated attack slices including leetspeak, Base64 payloads, and spaced text. - Latency Benchmarking: Measure P50, P95, and P99 inference latency in milliseconds with warmup runs. - Use Case: Compare a TF-IDF baseline against a fine-tuned DeBERTa-v3 classifier by running both through the robustness slices and confirming the false positive rate stays under 1.5% on benign prompts. ## Quick Start Evaluate my guardrail model's predictions against the test set and report F1, false positive rate, ROC-AUC, and per-slice adversarial robustness accuracy.

Frequently Asked Questions about guardrail-evaluation-metrics

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I calculate false positive rate for a prompt injection classifier?▼

Compute FPR as FP divided by (FP + TN) from the confusion matrix of your predictions. Use sklearn's confusion_matrix to extract true negatives and false positives, then divide to get the percentage of benign prompts mistakenly blocked.

How to benchmark LLM guardrail inference latency in Python?▼

Run warmup iterations first, then time each prediction with time.perf_counter and convert to milliseconds. Report P50, P95, and P99 percentiles with numpy to verify targets like P95 under 30ms per request.

What metrics should I use to evaluate a prompt injection detection model?▼

Use precision, recall, F1-score, false positive rate, and ROC-AUC from scikit-learn. Recall measures attack detection coverage while FPR measures over-blocking of benign prompts, and both must be balanced for guardrail evaluation.

How do I test a classifier against obfuscated attacks like leetspeak or Base64?▼

Build separate test slices for each obfuscation type such as leetspeak, Base64 payloads, and spaced text, then run the model's predict function on each slice. Compare per-slice accuracy to identify which evasion techniques bypass detection.

Does scikit-learn support ROC-AUC for binary guardrail classification?▼

Yes, roc_auc_score computes the area under the ROC curve when you pass true labels and predicted probabilities. Pass probability scores rather than hard class labels to get a threshold-independent ranking quality measure.

Related Skills

Explore More Data & Analytics Skills→
andikarachmanandikarachman
14

split-strategy

Select train/validation/test split strategies based on data characteristics.

Community
Intermediate
dengineproblemdengineproblem

ml-api-endpoint

Design and deploy FastAPI endpoints for machine learning model serving and inference.

Community
Advanced
aden-hiveaden-hive
11.0k

hive.quality-monitor

Periodically self-assess agent output quality to detect degradation during long-running tasks.

Official
Basic
DoanNgocCuongDoanNgocCuong

prompt-guard

Detect prompt injections and jailbreak attempts in LLM applications.

Community
Advanced