qwen_training_data_miner_prototype

To mine domain-specific training data for Gemma instruction-tuning, load your source file like 012.txt and specify the target domain alongside pattern types. The miner extracts relevant examples and outputs structured JSON datasets with corresponding pattern summaries.

Supported domains for extracting instruction-tuning datasets include mps_scoring, wsp_application, roadmap_analysis, readme_patterns, modlog_updates, and first_principles. The miner scans your source text to isolate these specific domain patterns for training.

You configure training example extraction by setting inputs including source_file, domain, pattern_type, and min_examples. These parameters filter the text file to ensure the generated instruction-tuning datasets meet your specific minimum example thresholds.

Mining domain patterns generates two primary outputs: a JSON file containing the instruction-tuning dataset saved to the training_datasets directory, and a pattern summary JSON file. An execution_id is also returned for full traceability.

While the miner is designed to produce instruction-tuning datasets for Gemma models, the generated JSON training data and pattern summaries are structured generically. You can adapt these outputs for other instruction-tuning workflows as needed.

Yes, the domain data extraction process lists pattern_memory and libido_monitor as dependencies. These modules support the underlying pattern recognition and monitoring required to parse source files and generate accurate training datasets.