What problem does it solve?
Manually managing complex financial data ingestion, feature engineering, and caching for the ATFT-GAT-FAN model is prone to errors, API quota issues, and training stalls. This Skill automates the entire data pipeline, ensuring data quality and availability.
Core Features & Use Cases
- Automated Dataset Provisioning: Generate fresh or historical Parquet datasets with GPU-accelerated ETL, ensuring data readiness for model training.
- Deterministic Feature Graph Management: Maintain hundreds of engineered factors with high determinism, crucial for consistent model accuracy.
- Use Case: Before a new training run, use this Skill to automatically refresh the 5-year dataset, verify cache integrity, and ensure all J-Quants API quotas are respected, preventing any data-related training interruptions.
Quick Start
Example: Refresh dataset in background and monitor
make dataset-bg
tail -f _logs/dataset/*.log