Deduplicate large ML training datasets using MinHash and LSH clustering.
npx skills add https://github.com/gar-ai/mallorn --skill mithril-dedup-agent
Please help me install this Agent Skill. Skill: mithril-dedup-agent Source: https://github.com/gar-ai/mallorn/tree/main/.claude/skills/mithril-dedup-agent Command: npx skills add https://github.com/gar-ai/mallorn --skill mithril-dedup-agent
Builds mithril-dedup for ML dataset deduplication, enabling scalable MinHash, LSH, clustering, and document I/O tooling to clean large training sets efficiently.
Run mithril-dedup on your dataset to identify near-duplicate documents and prepare a deduplicated output.
High-intent search queries and answers about installing and using this skill.
To deduplicate large ML training datasets, this skill uses MinHash and LSH to identify near-duplicate documents and clusters them with Union-Find. It processes large text collections to clean training sets efficiently.
Clustering similar documents in ML data cleaning is best handled by generating MinHash signatures, creating candidate pairs with Locality-Sensitive Hashing, and grouping duplicates using Union-Find clustering.
Yes, you can use this dataset deduplication tool with JSONL and Parquet formats. It provides built-in document I/O support to read and write these specific file formats during the cleaning workflow.
MinHash and LSH work for removing near-duplicate text by generating compact signature representations of documents and hashing them into buckets so similar items collide, creating candidate pairs for clustering.
No, you do not need external dependencies to run MinHash deduplication on your dataset. The skill operates independently with its own minhash, lsh, cluster, and io components accessible via a CLI.
thisismynewfmail-uiCurate multimodal datasets with quality filtering, deduplication, and PII redaction.
patricio0312revConvert documents into chunked JSONL datasets for embeddings.
aayushsoamAutomate multi-modal data curation for LLM training with deduplication and filtering.
sovr610Deduplicate training corpora and detect holdout leakage with MinHash+LSH.
