atft-pipeline

Provision fresh or historical Parquet datasets for ATFT-GAT-FAN using GPU-accelerated ETL.

Updated Aug 28, 2025
One-click install
npx skills add https://github.com/wer-inc/gogooku3 --skill atft-pipeline
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: atft-pipeline
Source: https://github.com/wer-inc/gogooku3/tree/main/claude/skills/atft-pipeline
Command: npx skills add https://github.com/wer-inc/gogooku3 --skill atft-pipeline

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires polars, pyarrow, cudf, jquants-api-client, and includes scripts (resource) components.

What problem does it solve?

Manually managing complex financial data ingestion, feature engineering, and caching for the ATFT-GAT-FAN model is prone to errors, API quota issues, and training stalls. This Skill automates the entire data pipeline, ensuring data quality and availability.

Core Features & Use Cases

  • Automated Dataset Provisioning: Generate fresh or historical Parquet datasets with GPU-accelerated ETL, ensuring data readiness for model training.
  • Deterministic Feature Graph Management: Maintain hundreds of engineered factors with high determinism, crucial for consistent model accuracy.
  • Use Case: Before a new training run, use this Skill to automatically refresh the 5-year dataset, verify cache integrity, and ensure all J-Quants API quotas are respected, preventing any data-related training interruptions.

Quick Start

Example: Refresh dataset in background and monitor

make dataset-bg tail -f _logs/dataset/*.log

Frequently Asked Questions about atft-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate financial data ETL for machine learning model training?

Financial data ETL automates ingestion, feature engineering, and caching of datasets for model training. This Skill provisions fresh or historical Parquet datasets with GPU acceleration, applies deterministic feature graph generation, and manages cache to prevent training stalls and API quota issues.

Can I use GPU acceleration to speed up feature engineering for large financial datasets?

Yes. This Skill uses GPU-accelerated ETL via cuDF and PyArrow to process large financial datasets efficiently. GPU acceleration speeds feature computation while maintaining deterministic results across hundreds of engineered factors for consistent model accuracy.

How do I refresh and validate datasets before training without manual data pipeline management?

Run automated dataset provisioning to generate fresh or historical Parquet data, verify cache integrity, and validate GPU availability and credentials. The Skill includes health checks and observability to ensure reliable, auditable dataset builds with clear logging.

What's the best way to prevent API quota exhaustion and data staleness during continuous model training?

Cache hygiene and deterministic feature graph management prevent quota overages and training interruptions. This Skill monitors API usage, maintains cache state across pipeline runs, and ensures datasets stay fresh and ready without manual intervention.

Does this work with J-Quants financial data and Parquet output formats?

Yes. The Skill integrates with J-Quants API client to fetch financial data and outputs deterministic Parquet datasets. It handles J-Quants quota management and applies feature engineering specifically for ATFT-GAT-FAN model requirements.

What dependencies and environment setup do I need before running the pipeline?

Required dependencies are Polars, PyArrow, cuDF, and jquants-api-client. The Skill validates GPU availability, API credentials, and all dependencies during setup, performing health checks to ensure the environment is ready before dataset builds begin.