skills.rest
ricardolui
ricardoluiCOMMUNITY
Explore all skills by this author →
GitHub

flink-udf

Build and deploy Java user-defined functions for Apache Flink stream processing over Kafka.

Updated Jul 7, 2026
Search Tags:#docker#java#kafka#stream processing#flink#udf#confluent cloud
One-click install
npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill flink-udf-ricardolui
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flink-udf
Source: https://github.com/ricardolui/gcp-custom-agent-skills/tree/main/flink-udf
Command: npx skills add https://github.com/ricardolui/gcp-custom-agent-skills --skill flink-udf-ricardolui

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Extending Flink SQL and the Table API with custom business logic requires writing, packaging, and deploying Java functions, which involves complex build configuration, state management, and platform-specific deployment steps that are easy to get wrong. ## Core Features & Use Cases - Three Function Types: Generate scalar UDFs, user-defined table functions (UDTFs), and stateful process table functions (PTFs) with proper annotations like @StateHint and @ArgumentHint. - Dual Deployment Targets: Deploy to Confluent Cloud via artifact upload or to local Docker environments via docker cp and ADD JAR. - Guided Workflow: Routes through infrastructure setup, Maven/Gradle build configuration, function registration, and testing with sample data. - Use Case: Build a PTF that detects duplicate events within a one-hour window per user, package it as a JAR, upload it to Confluent Cloud, and invoke it from Flink SQL with PARTITION BY and a stable uid for state recovery. ## Quick Start Ask the assistant to create a Flink scalar UDF in Java that extracts the domain from email addresses and deploy it to Confluent Cloud for use in Flink SQL.

Frequently Asked Questions about flink-udf

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a Flink UDF in Java and deploy it to Confluent Cloud?▼

Extend ScalarFunction with an eval() method, build a JAR with Maven or Gradle using provided-scope Flink dependencies, then upload it with confluent flink artifact create. Register it in Flink SQL using CREATE FUNCTION with the USING JAR 'confluent-artifact://' clause.

What is the difference between a Flink scalar UDF, UDTF, and PTF?▼

A scalar UDF maps input values to one output value, a UDTF emits multiple output rows per input via TableFunction and collect(), and a PTF adds managed state, partitioning, and timers for N-to-M stateful processing like deduplication or sessionization.

How do I deploy a Flink UDF to a local Docker environment?▼

Build the JAR, copy it into the flink-sql-client container with docker cp, then load it at runtime with ADD JAR and register it using CREATE FUNCTION. Use broker:9092 as the Kafka bootstrap server inside Docker and localhost:29092 from external Table API clients.

Why does my Flink PTF fail with a uid or PARTITION BY error?▼

PTFs require a PARTITION BY clause to define state partitioning and a stable uid parameter for state recovery across restarts. Add PARTITION BY on your partition key and pass uid => 'my-ptf-v1' in the invocation, keeping the uid stable across deployments.

Should Flink dependencies be marked as provided in the Maven pom.xml?▼

For Confluent Cloud, mark Flink dependencies as provided since the platform supplies them at runtime, and exclude them from the shaded JAR. For local Docker deployments, bundle Flink dependencies into the JAR because the local runtime does not provide all required modules.

Related Skills

Explore More Software Engineering Skills→
confluentincconfluentinc
42

developing-kafka-python-client

Create a Python Kafka client project with Schema Registry and JSON Schema validation.

Official
Intermediate
Samuelca6399Samuelca6399

real-time-streaming

Design and operate Kafka and Flink pipelines with exactly-once processing.

Community
Advanced
kalilurrahmankalilurrahman
1

streaming-pipeline

Design real-time streaming pipelines with Kafka, Flink, and Spark Streaming.

Community
Advanced
AbsolutelySkilledAbsolutelySkilled
207

real-time-streaming

Guide building and operating Kafka and Flink streaming pipelines with exactly-once semantics.

Official
Advanced