skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

apache-flink

Design Apache Flink streaming pipelines with event-time windowed aggregations and exactly-once semantics.

14|1|Updated May 5, 2026
Search Tags:#streaming#kafka#windowing#watermarks#exactly-once#apache flink#pyflink
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill apache-flink
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: apache-flink
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/skills/apache_flink
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill apache-flink

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Apache Flink streamlines building reliable, low-latency data pipelines that process events in motion, handle out-of-order data, and recover safely from failures without losing or duplicating records.

Core Features & Use Cases

  • Event-time streaming with windowing: TUMBLE/HOP/SESSION/CUMULATE window definitions using watermarks for correct results under out-of-order arrivals.
  • Stateful processing at scale: Keyed operations with managed state (e.g., ValueState, MapState) and timer-based session/dedup logic using the DataStream API.
  • Production-grade fault tolerance: Checkpointing with EXACTLY_ONCE semantics, configurable state backends (HashMap vs RocksDB), and savepoints for controlled upgrades.

Quick Start

Load the apache-flink skill and ask it to generate a PyFlink SQL streaming job that reads events from Kafka, applies an event-time watermark, performs a 5-minute tumbling aggregation, and writes results to Kafka with EXACTLY_ONCE delivery.

Frequently Asked Questions about apache-flink

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a streaming pipeline that handles out-of-order data with exactly-once semantics?▼

Apache Flink streaming pipelines handle out-of-order data using event-time watermarks and windowed aggregations, while checkpointing with RocksDB state backend ensures exactly-once semantics and safe failure recovery without losing or duplicating records.

How does event-time windowing work for Kafka ETL jobs?▼

Event-time windowing in Kafka ETL jobs groups records into TUMBLE, HOP, SESSION, or CUMULATE windows based on event timestamps, using watermarks to correctly process delayed events before triggering aggregation results.

Can I use PyFlink Table API and SQL for real-time analytics and deduplication?▼

PyFlink supports both Table API and SQL for real-time analytics and deduplication, allowing you to define streaming queries over Kafka sources with event-time watermarks and exactly-once Kafka connector delivery guarantees.

What is the best way to configure fault tolerance and state management for stateful streaming applications?▼

Configuring fault tolerance for stateful streaming requires enabling EXACTLY_ONCE checkpointing and selecting a state backend like RocksDB, which allows managed state such as ValueState and MapState to scale while supporting savepoints for controlled upgrades.

Does Apache Flink support CDC ingestion and session detection in a single data pipeline?▼

Apache Flink supports CDC ingestion and session detection within a single pipeline by applying keyed operations with managed state and timer-based logic to identify user sessions and process change events from filesystem or Kafka sources.

Related Skills

Explore More Software Engineering Skills→
AbsolutelySkilledAbsolutelySkilled
207

real-time-streaming

Guide building and operating Kafka and Flink streaming pipelines with exactly-once semantics.

Official
Advanced
ice-droid-99ice-droid-99

databricks-spark-structured-streaming

Design Spark Structured Streaming pipelines ingesting Kafka with checkpoint recovery.

Community
Advanced
ivanshamaevivanshamaev
14

pyspark-structured-streaming

Build PySpark Structured Streaming pipelines for Kafka, file, and rate sources.

Community
Advanced
manutejmanutej
61

kafka-stream-processing

Implement Apache Kafka producers, consumers, streams, and connectors for real-time data processing.

Community
Advanced