ml-pipeline-architect

Design automated ML training pipelines on Kubernetes with Argo Workflows or Kubeflow.

Updated Jul 16, 2026
One-click install
npx skills add https://github.com/Cloud-Byte-Consulting/plugins --skill ml-pipeline-architect-cloud-byte-consulting
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ml-pipeline-architect
Source: https://github.com/Cloud-Byte-Consulting/plugins/tree/main/model-training-ops/skills/ml-pipeline-architect
Command: npx skills add https://github.com/Cloud-Byte-Consulting/plugins --skill ml-pipeline-architect-cloud-byte-consulting

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Teams running one-off training scripts lack repeatable, automated ML pipelines, leading to stale models, bad-data incidents reaching production, and uncontrolled GPU costs on Kubernetes. ## Core Features & Use Cases - Pipeline Step Design: Instantiate a full step taxonomy (data validation, feature engineering, training, evaluation, conditional registration) with typed inputs, outputs, and quality gates. - Orchestrator & Trigger Selection: Choose between Argo Workflows, Kubeflow Pipelines, Airflow, or TFX using explicit decision rules, and pick from four trigger types (GitOps commit, new-data event, schedule, statistical drift). - Cost Engineering: Apply step caching keyed on dataset versions and spot/preemptible GPU training with mandatory checkpoint-and-resume logic. - Use Case: A research team on managed Kubernetes wants retraining to fire automatically when new data lands in S3-compatible storage; the Skill produces a pipeline design doc with a trigger spec, caching plan, and a skeleton Argo WorkflowTemplate. ## Quick Start Use the ml-pipeline-architect skill to design an automated retraining pipeline on Argo Workflows that triggers on new data in our object store and registers models in MLflow only above a metric threshold.

Frequently Asked Questions about ml-pipeline-architect

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate ML model retraining on Kubernetes?

Define a pipeline with data validation, training, evaluation, and conditional registration steps, then attach a trigger such as a GitOps commit, S3 bucket event, CronWorkflow schedule, or drift alert. Argo Workflows or Kubeflow Pipelines executes the DAG with each step as a pod.

Argo Workflows vs Kubeflow Pipelines for ML training?

Argo Workflows is the default for Kubernetes-first shops needing general-purpose orchestration with minimal overhead. Choose Kubeflow Pipelines only if the team will use at least two other Kubeflow components like Notebooks, Katib, or KServe, since it is a platform to operate.

Can I use spot or preemptible GPUs for training pipelines?

Yes, but only with mandatory checkpointing to object storage so replacement pods resume from the last checkpoint after preemption. Cap retries, make steps idempotent, and keep orchestration and short critical steps on on-demand nodes.

How does step caching work in ML pipelines?

The orchestrator skips a step when its cache key of input artifacts, parameters, and image matches a prior run. Key caches on dataset version rather than file path, set a TTL, and never cache the evaluation or condition gate.

When should I not build an ML training pipeline?

Avoid pipelines during rapid feature, model, or hyperparameter exploration, where pipeline rigidity slows iteration. Deploy pipelines once training becomes regular, and do not add drift-triggered retraining before a working manually-started pipeline exists.