skills.rest
ice-droid-99
ice-droid-99COMMUNITY
Explore all skills by this author →
GitHub

databricks-spark-structured-streaming

Design Spark Structured Streaming pipelines ingesting Kafka with checkpoint recovery.

Updated May 20, 2026
Search Tags:#databricks#watermarks#spark structured streaming#kafka ingestion#checkpoint best practices#stream joins#real-time mode rtm
One-click install
npx skills add https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks- --skill databricks-spark-structured-streaming-ice-droid-99
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: databricks-spark-structured-streaming
Source: https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks-/tree/main/.gemini/skills/databricks-spark-structured-streaming
Command: npx skills add https://github.com/ice-droid-99/BI-Dashboard-Generation-Agent-in-Databricks- --skill databricks-spark-structured-streaming-ice-droid-99

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Spark Structured Streaming setups often fail in production due to missing checkpoint discipline, incorrect trigger choices, unstable stateful logic, and unsafe join/write patterns that break exactly-once or cause data loss. This Skill provides production-oriented guidance to design streaming pipelines that remain correct under restarts, late data, and scaling changes.

Core Features & Use Cases

  • Production-ready streaming patterns: Covers Kafka ingestion, RTM vs micro-batch execution, stateful operations with watermarks, and stream-stream/stream-static join strategies.
  • Reliability and correctness: Emphasizes checkpoint best practices and exactly-once verification guidance for robust recovery scenarios.
  • Performance and cost tuning: Guides trigger selection (processingTime, availableNow, RTM) and operational configuration for balancing latency vs cost in Databricks.
  • Practical integration targets: Common targets include Delta sinks, multi-sink fan-out, and enrichment/merge patterns that support real-world BI/ETL flows.

Quick Start

Use the databricks-spark-structured-streaming skill to implement a production-grade Kafka-to-Delta streaming job with a persistent checkpoint in Unity Catalog Volumes and a configured processing-time trigger.

Frequently Asked Questions about databricks-spark-structured-streaming

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I configure watermarks and checkpoint locations for Spark Structured Streaming pipelines?▼

Spark Structured Streaming pipelines require persistent unique checkpoint locations and proper watermark handling to ensure correct operational behavior during stateful operations and recoveryrestarts. This Skill guides production-grade configuration for these exact scenarios.

What is the best way to ingest Kafka data into Delta sinks using Spark Structured Streaming?▼

Ingesting Kafka data into Delta sinks requires production-oriented streaming patterns that support reliable multi-sink fan-out. This Skill provides guidance for configuring Kafka ingestion and writing to reliable sinks using Databricks.

Does this Skill support real-time mode (RTM) and availableNow triggers for Databricks streaming jobs?▼

Yes, this Skill guides trigger selection between processingTime, availableNow, and real-time mode (RTM) for Databricks streaming jobs. It balances latency versus cost by configuring operational execution modes for near-real-time workloads.

Can I perform stream-stream and stream-static joins in Spark Structured Streaming without breaking exactly-once semantics?▼

Performing stream-stream and stream-static joins requires streaming-safe join patterns to prevent breaking exactly-once semantics. This Skill provides strategies for enrichment and merge patterns while maintaining production reliability.

Why does my Spark Structured Streaming job fail during restarts or scaling changes?▼

Streaming jobs often fail during restarts and scaling changes due to missing checkpoint discipline and unstable stateful logic. This Skill ensures pipeline correctness by enforcing persistent checkpoint locations and proper state management.

Related Skills

Explore More Data & Analytics Skills→
PaldomPaldom
11

databricks-synthetic-data-gen

Generate synthetic data with Spark and Faker into Unity Catalog volumes.

Community
Advanced
XCmiaowXCmiaow

data-analytics

Generate PlantUML ETL and data analytics architecture diagrams from textual pipeline descriptions.

Community
Intermediate
chicanoandres702chicanoandres702

data-engineering-data-pipeline

Design and orchestrate scalable batch and streaming data pipelines.

Community
Advanced
m19cm19c

databricks-spark-structured-streaming

Build and optimize Spark Structured Streaming pipelines with Kafka integration.

Community
Advanced