data-pipeline

To build a pretraining corpus for LLMs, you can automate the ingestion of raw scraped web data and apply cleaning, MinHash deduplication, heuristic quality filtering, and PII removal to ensure dataset safety.

MinHash deduplication is a data cleaning technique that efficiently identifies and removes near-duplicate text documents. It ensures data uniqueness in large-scale LLM pretraining corpora by estimating document similarity.

To remove PII from a dataset, apply automated PII redaction during the data pipeline process. This ensures sensitive personal information is scrubbed from documents before pretraining an LLM.

Yes, you can use pandas and pyarrow to process large datasets for LLM training. The pipeline leverages these libraries to handle data manipulation and columnar storage efficiently during automated corpus creation.

Yes, the data pipeline supports FineWeb-Edu style quality filtering. It applies heuristic-based filters to retain high-quality text documents, ensuring balanced domain representation for pretraining.

Using regex and ftfy for text cleaning handles encoding fixes and pattern removal, but may struggle with highly unstructured web data. Additional pipeline steps like quality filtering are needed to ensure corpus integrity.