dataset-discovery

Discover ML datasets across HuggingFace, OpenML, GitHub, and papers using multi-source search. The workflow aggregates results, deduplicates entries, and ranks datasets by metadata relevance to return a consolidated list.

Find datasets for text classification or image classification by specifying your task during the multi-source discovery process. The search targets relevant datasets across modalities like text, image, and tabular data, returning ranked candidates for rapid experimentation.

Preview dataset samples and metadata using the optional detail and sample pull workflows. You can fetch specific dataset metadata and inspect sample rows directly to determine suitability before downloading.

You need to install the requests dependency to search for ML datasets across HuggingFace and GitHub. The scripts component requires this library to execute multi-source search and fetch dataset metadata over HTTP.

Cross-source deduplication works by matching dataset metadata across HuggingFace, OpenML, GitHub, and papers during aggregation. It removes duplicate entries to ensure the final ranked list contains unique datasets for your ML experiments.

Ranked dataset search results are output in markdown and JSON artifact formats. The markdown provides readable documentation while the JSON artifacts enable programmatic integration of the deduplicated dataset metadata into downstream workflows.