aiml-detoxify-benchmark

Automate cross-category toxicity benchmarking for Detoxify classifiers.

934|152|Updated Mar 1, 2026

Search Tags:#benchmark #ai-safety #toxicity #ml-evaluation #toxicity-benchmark #detoxify

Ready to run

npx skills add https://github.com/wuyoscar/ISC-Bench --skill aiml-detoxify-benchmark

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: aiml-detoxify-benchmark
Source: https://github.com/wuyoscar/ISC-Bench/tree/main/templates/aiml_detoxify
Command: npx skills add https://github.com/wuyoscar/ISC-Bench --skill aiml-detoxify-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Automates cross-category toxicity benchmarking for Detoxify-based classifiers.

Core Features & Use Cases

Provides a repeatable evaluation harness for toxicity detection across six Jigsaw categories (toxic, severe_toxic, obscene, threat, insult, identity_hate) and styled variants.
Validates dataset quality by checking placeholders, minimum text length, and category coverage.
Generates scores and recall metrics to support model reliability assessments in safety-critical deployments.

Quick Start

Run the Detoxify benchmark against your model to generate recall results for all categories.

Frequently Asked Questions about aiml-detoxify-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I benchmark a Detoxify classifier for toxicity detection across multiple categories?▼

To benchmark Detoxify toxicity detection, provide a test dataset with non-placeholder text and a Detoxify model to score predictions. The benchmark script automates cross-category evaluation to generate scores and recall metrics for six toxicity categories.

What toxicity categories are evaluated in a Detoxify benchmark?▼

A Detoxify benchmark evaluates six Jigsaw toxicity categories: toxic, severe_toxic, obscene, threat, insult, and identity_hate. It assesses model performance across these categories and their styled variants to generate recall metrics.

What are the dataset requirements for running toxicity benchmarking on Detoxify models?▼

Toxicity benchmarking requires a test dataset with non-placeholder text and minimum length thresholds. The benchmark validates dataset quality by checking for placeholders, minimum text length, and category coverage before running evaluations.

How does a recall-based toxicity benchmark support AI safety evaluations?▼

A recall-based toxicity benchmark supports AI safety evaluations by generating scores and recall metrics across six toxicity categories. This repeatable evaluation harness validates model performance and reliability for safety-critical deployments.

Can I use this benchmark for ML evaluation on styled variants of toxic text?▼

Yes, the benchmark automates ML evaluation for styled variants of toxic text. It applies a repeatable evaluation harness to Detoxify-based classifiers to validate model performance across six Jigsaw toxicity categories and their styled variants.

Why does my toxicity benchmark fail to validate dataset quality?▼

A toxicity benchmark fails dataset quality validation if the test data contains placeholders, does not meet minimum text length thresholds, or lacks category coverage. The benchmark script checks these conditions before scoring predictions with the Detoxify model.