skills.rest
bbvch-ai
bbvch-aiOFFICIAL
Explore all skills by this author →
GitHub

debug-pipeline

Diagnose Dagster pipeline failures using Dagster MCP tools.

9|2|Updated Nov 29, 2024
Search Tags:#s3#dagster#milvus#ai-infrastructure#pipeline-debugging#asset-materialization#sensor-troubleshooting
One-click install
npx skills add https://github.com/bbvch-ai/aihub-core --skill debug-pipeline-bbvch-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: debug-pipeline
Source: https://github.com/bbvch-ai/aihub-core/tree/main/.claude/skills/debug-pipeline
Command: npx skills add https://github.com/bbvch-ai/aihub-core --skill debug-pipeline-bbvch-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Dagster pipeline failures—including asset materialization errors, sensor malfunctions, partition issues, and connection problems with integrated services like S3, Milvus, and rclone—cause costly downtime for AI infrastructure stacks and require hours of manual troubleshooting to resolve.

Core Features & Use Cases

  • Direct Instance Inspection: Uses Dagster MCP tools to pull run logs, asset status, and error details directly from the running Dagster instance, eliminating the need to manually navigate the UI.
  • Infrastructure Troubleshooting: Provides step-by-step guidance to fix connection and configuration issues for core services including S3/MinIO, Milvus, MongoDB, rclone, and NATS.
  • Pre-Defined Error Fixes: Includes solutions for common failure patterns like partition stuck states, embedding failures, document parsing errors, and IO manager type mismatches.
  • Use Case: If your document ingestion pipeline fails due to a Milvus connection timeout, this skill walks you through verifying Milvus is running, checking embedding dimension settings, and re-triggering the failed asset run.

Quick Start

Use the debug-pipeline skill to investigate why your document ingestion asset is failing to materialize and resolve the root cause.

Frequently Asked Questions about debug-pipeline

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I debug Dagster asset materialization failures in an AI pipeline?▼

To debug Dagster asset materialization failures, you inspect run logs, asset status, and infrastructure configurations directly from the running instance to identify root causes and apply remediation steps for the failed AI workload.

Why does my Dagster sensor malfunction and fail to trigger pipeline runs?▼

Dagster sensor malfunctions often stem from misconfigured settings or connection issues with integrated services. Troubleshooting involves inspecting sensor logs and infrastructure configurations to identify the root cause and restore proper triggering.

How do I fix a Milvus connection timeout during document ingestion?▼

Fixing a Milvus connection timeout involves verifying that the Milvus service is running, checking embedding dimension settings to ensure they match, and re-triggering the failed asset run after resolving the connection issue.

What is the best way to resolve stuck partition states in Dagster?▼

The best way to resolve stuck partition states in Dagster is to inspect the run logs and asset status for the affected partitions, identify the specific error pattern, and apply predefined fixes to re-materialize the partition.

Can I troubleshoot S3 and NATS resource misconfigurations without navigating the Dagster UI?▼

Yes, you can troubleshoot S3, NATS, and other resource misconfigurations without the UI by using Dagster MCP tools to pull error details, run logs, and infrastructure configurations directly from the running Dagster instance.

How do I fix IO manager type mismatches when materializing Dagster assets?▼

Fixing IO manager type mismatches requires inspecting the asset error details to confirm the mismatch, correcting the resource configuration for the IO manager, and re-triggering the failed asset materialization run.

Related Skills

Explore More Software Engineering Skills→
i3ringiti3ringit
3

rclone

Automate cross-cloud file transfers and synchronization using rclone.

Community
Advanced
ivanshamaevivanshamaev
14

dagster-assets

Define and automate Dagster asset materialization with dependencies and partitions.

Community
Advanced
bbvch-aibbvch-ai
9

scaffold-pipeline

Generate Dagster pipeline boilerplate for external data sources and custom processing extensions.

Official
Intermediate
zilliztechzilliztech
3

local-setup

Deploy local Milvus environments using Milvus Lite, Docker Standalone, or Docker Compose.

Official
Advanced