nemo-curator

GPU-accelerated data curation for LLM training datasets uses RAPIDS and Dask to scale processing across GPUs, achieving up to 16x faster deduplication. It supports exact, fuzzy, and semantic matching to clean massive text collections efficiently.

GPU-accelerated data curation supports multimodal formats including text, image, video, and audio. It applies quality filtering and deduplication across these diverse data types to prepare comprehensive foundation model training sets.

Semantic deduplication in LLM data curation identifies and removes contextually similar data points rather than exact matches. This ensures higher dataset diversity by leveraging GPU acceleration for scalable processing across large text collections.

Yes, nemo-curator requires a GPU for text processing as it relies on RAPIDS and cudf for GPU acceleration. You must install the necessary libraries using the command `uv pip install "nemo-curator[text_cuda12]"`.

To redact PII and detect NSFW content in training data, use GPU-accelerated data curation tools that apply these filters at scale. This ensures sensitive information is removed during the dataset preparation process for foundation models.

The best way to curate massive datasets like 10TB of text and images is using GPU-accelerated tools with RAPIDS and Dask. This approach handles quality filtering, deduplication, and PII redaction efficiently across distributed GPUs.