skills.rest
cas-bigdatalab
cas-bigdatalabOFFICIAL
Explore all skills by this author →
WebsiteGitHub

sink_stop

Terminate a Dataflow Graph at a specified node and save output to a path.

541|171|Updated May 3, 2018
Search Tags:#data processing#data storage#dag#dataflow#output termination
One-click install
npx skills add https://github.com/cas-bigdatalab/piflow --skill sink-stop
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sink_stop
Source: https://github.com/cas-bigdatalab/piflow/tree/main/workspace/dag_system_node/sink_stop
Command: npx skills add https://github.com/cas-bigdatalab/piflow --skill sink-stop

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill addresses the need to terminate a Dataflow Graph (DAG) at a specific node, ensuring that the output is saved to a designated location without further processing.

Core Features & Use Cases

  • DAG Termination: Stops the execution of a DAG at the sink_stop node.
  • Output Saving: Saves the output data from the preceding nodes to a specified path.
  • Use Case: Ideal for scenarios where you want to ensure that a particular stage of a DAG is complete and the data is safely stored before proceeding with other workflows.

Quick Start

Execute the sink_stop skill with the 'input' parameter set to the output of the previous node and the 'path' parameter set to the desired output file location.

Frequently Asked Questions about sink_stop

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I terminate a DAG at a specific node and save the output?▼

To terminate a DAG at a specific node and save the output, you can use the sink_stop function. It stops Dataflow Graph execution at the designated node and writes the preceding output data to a specified file path.

When do I need to stop dataflow graph execution early for data storage?▼

You need to stop dataflow graph execution early for data storage when a particular DAG stage must be completed and safely stored before proceeding with other workflows. This ensures intermediate data is preserved without further processing.

What is the best way to save intermediate results from a complex data processing workflow?▼

The best way to save intermediate results from a complex data processing workflow is to terminate the DAG at the desired stage. This captures the output from preceding nodes and saves it directly to a designated path.

How do I configure the input and output path for a DAG sink node?▼

To configure the DAG sink node, set the input parameter to the output of the previous node and the path parameter to your desired output file location. This routes the dataflow graph output to the specified storage path.

Can I halt a dataflow graph without losing the data from preceding nodes?▼

Yes, you can halt a dataflow graph without losing data by using a DAG termination node. It stops graph execution while saving the output data from all preceding nodes to a designated storage path.

Related Skills

Explore More Software Engineering Skills→
miptah21miptah21
2

tracing-upstream-lineage

Trace upstream data lineage for Airflow tables, columns, and DAGs.

Community
Intermediate
cas-bigdatalabcas-bigdatalab
541

source_stop

Configure the initial data source for a DAG workflow with a file path.

Official
Basic
tomztomz

gcp-data

Configure Google Cloud data services for scalable data platforms using gcloud CLI.

Community
Advanced
bettercallfanbettercallfan

pipeline-graph-builder

Construct business-level DAGs from code analysis results using Python and NetworkX.

Community
Advanced