What problem does it solve?
Nemo Curator addresses the challenge of preparing high-quality training data for Large Language Models (LLMs) by offering fast and efficient data curation features, including deduplication, quality filtering, and redaction.
Core Features & Use Cases
- Multi-Modal Curation: Supports text, image, video, and audio data types for LLM training.
- Fuzzy Deduplication: 16× faster than CPU for deduplication, handling large corpora with precision.
- Quality Filtering: Employ over 30 heuristic filters for quality assurance and data cleanup.
- Semantic Deduplication: Remove semantically similar documents with high accuracy.
- PII Redaction: Safely redact sensitive personal information (PII) to ensure privacy.
- NSFW Detection: Identifies and filters out inappropriate content.
- Use Case: Utilize Nemo Curator to curate large datasets for training NLP models, enhancing their accuracy and reducing bias.
Quick Start
To start using Nemo Curator, first install it with uv pip install nemo-curator. Then, run a simple text curation pipeline like this:
python
from nemo_curator import ScoreFilter, Modify
from nemo_curator.datasets import DocumentDataset
import pandas as pd
# Load data
df = pd.DataFrame({"text": ["Good document", "Bad doc", "Excellent text"]})
dataset = DocumentDataset(df)
# Quality filtering
def quality_score(doc):
return len(doc["text"].split()) > 5 # Filter short docs
filtered = ScoreFilter(quality_score)(dataset)
# Deduplication
from nemo_curator.modules import ExactDuplicates
deduped = ExactDuplicates()(filtered)
# Save
deduped.to_parquet("curated_data/")
