skills.rest
IBM
IBMOFFICIAL
Explore all skills by this author →
Website𝕏@ibmdeveloperGitHub

di-agent-flow-substrait-datastage

Generate a DataStage flow and Python SDK script from a Substrait plan.

3|Updated May 1, 2026
Search Tags:#mcp#etl#data integration#python sdk#flow generation#substrait#datastage
One-click install
npx skills add https://github.com/IBM/ibm-watsonx-data-integration-skills --skill di-agent-flow-substrait-datastage
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: di-agent-flow-substrait-datastage
Source: https://github.com/IBM/ibm-watsonx-data-integration-skills/tree/main/agent/skills/di-agent-flow-substrait-datastage
Command: npx skills add https://github.com/IBM/ibm-watsonx-data-integration-skills --skill di-agent-flow-substrait-datastage

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

This Skill converts an optimized Substrait plan into a runnable IBM DataStage flow so an AI agent can produce production-ready data integration code without manually assembling stages.

Core Features & Use Cases

  • Full-pushdown generation: Builds a two-stage DataStage flow from a validated optimized plan with one SQL-mode source and one Sequential file sink.
  • Safe validation: Checks the plan shape, required connection metadata, output schema types, and supported connector behavior before generating code.
  • Flow lifecycle support: Produces both a complete Python SDK script and an MCP-ready code subset for creating, updating, and optionally running the flow.
  • Use Case: An agent receives a Substrait plan from query optimization and turns it into a DataStage job that executes the SQL in the source system and writes the results to a CSV file.

Quick Start

Ask the skill to generate a DataStage Python SDK script from your optimized full-pushdown Substrait plan and provide the matching DataStage connection name so it can build the flow.

Frequently Asked Questions about di-agent-flow-substrait-datastage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I convert an optimized Substrait plan into a DataStage flow?▼

To convert an optimized Substrait plan into a DataStage flow, provide the validated plan and matching DataStage connection name. The Skill generates a two-stage flow with a SQL-mode source and Sequential file sink.

Does DataStage flow generation support multi-source SQL pushdown or only single-source reads?▼

DataStage flow generation only supports single-source SQL pushdown reads. The Skill applies full-pushdown shape checks to ensure the optimized Substrait plan converts safely into a Sequential file sink.

What do I need to generate an MCP-ready DataStage Python SDK script?▼

You need a validated optimized Substrait plan, schema typing, connector-specific stage mapping, and DataStage connection metadata. The Skill uses these to produce MCP-ready Python SDK code for flow lifecycle management.

Can I create and run DataStage jobs directly from a Substrait plan using Python?▼

Yes, you can create, update, and optionally run DataStage jobs from a Substrait plan. The Skill generates a complete Python SDK script that handles the flow lifecycle and execution.

What are the limitations when generating DataStage flows from Substrait plans?▼

Limitations include requiring a full-pushdown plan shape, validated connection metadata, and supported connector behavior. The Skill performs safe validation checks before generating the DataStage flow and Python SDK script.

Related Skills

Explore More Data & Analytics Skills→
IBMIBM

di-agent-query-optimization

Rewrite Substrait plans into compact full-pushdown SQL for DataStage execution.

Official
Advanced
JingyiWngJingyiWng

databricks-jobs

Manage Databricks Jobs lifecycle via Python SDK, CLI, and Asset Bundles.

Community
Advanced
EmeaAppGbbEmeaAppGbb

microsoft-agent-framework

Build and orchestrate AI agents with Microsoft Agent Framework.

Official
Intermediate
IBMIBM

di-agent-knowledge-engine-datastage

Explain DataStage parallel engine concepts, stage semantics, and optimization guidance.

Official
Advanced