skills.rest
paul-heyse
paul-heyseCOMMUNITY
Explore all skills by this author →
GitHub

dfdl_ref

Document DataFusion and DeltaLake integration patterns for planning and execution workflows.

Updated Jan 10, 2026
Search Tags:#planning#python#sql#rust#datafusion#information_schema#deltalake
One-click install
npx skills add https://github.com/paul-heyse/CodeAnatomy --skill dfdl-ref
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dfdl_ref
Source: https://github.com/paul-heyse/CodeAnatomy/tree/main/.claude/skills/dfdl_ref
Command: npx skills add https://github.com/paul-heyse/CodeAnatomy --skill dfdl-ref

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Provide a comprehensive operations manual for DataFusion + DeltaLake integration, detailing how to wire the core query engine with the storage layer, including planning, pushdown, scan providers, and schema bridging, while guiding lookup patterns and avoiding API guesswork.

Core Features & Use Cases

  • In-depth coverage of DataFusion catalog/schema management, external tables, and predicate pushdown with DeltaLake integration nuances.
  • Practical guidance on programmatic plan construction, subqueries, and UDF usage across Rust and Python bindings, including how planning surfaces map to execution.
  • DeltaLake-specific integration notes (time travel, file pruning, MVCC log semantics) with example workflows for DataFusion planning, SQL/DDL usage, and data registration.

Quick Start

Load a DeltaTable into a DataFusion context and run a simple plan to observe scan pruning and predicate pushdown.

Frequently Asked Questions about dfdl_ref

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I integrate DeltaLake with DataFusion for query planning and execution?▼

Integrate DeltaLake with DataFusion by wiring the core query engine to the storage layer through catalog management, external tables, and scan providers. This bridges schemas and enables predicate pushdown for reproducible planning and execution workflows.

Does DataFusion support predicate pushdown and file pruning with DeltaLake tables?▼

DataFusion supports predicate pushdown and file pruning with DeltaLake tables through delta-aware data sources. Loading a DeltaTable into a DataFusion context allows the planner to observe scan pruning and apply predicates directly during execution.

Can I use DataFusion catalog and schema management with external DeltaLake tables in Python and Rust?▼

DataFusion provides catalog and schema management for external DeltaLake tables across both Python and Rust bindings. You can register delta tables, handle lookups, and manage external table integration without API guesswork.

How do I handle Delta transaction semantics and time travel in DataFusion planning workflows?▼

Handle Delta transaction semantics in DataFusion by leveraging MVCC log semantics and time travel capabilities. The integration provides specific patterns for managing delta-aware data sources and ensuring reproducible plan serialization across queries.

What is the best way to construct programmatic query plans with subqueries and UDFs in DataFusion?▼

Construct programmatic query plans in DataFusion by using the planning surfaces that map to execution across Rust and Python bindings. The integration supports subquery construction, UDF usage, and SQL/DDL workflows for complex analytics.

Why does my DataFusion DeltaLake integration require information_schema lookups during planning?▼

DataFusion DeltaLake integration requires information_schema lookups during planning to bridge schemas between the query engine and storage layer. Proper catalog management ensures external tables are registered correctly for cross-engine planning.

Related Skills

Explore More Data & Analytics Skills→
apacheapache
595

check-upstream

Audits python-api coverage of upstream DataFusion Rust bindings and reports gaps.

Official
Intermediate
gnestorgnestor

google-bigquery

Translate natural language questions into BigQuery SQL and execute them.

Community
Advanced
89jobrien89jobrien
4

database-optimization

Optimize slow SQL queries and resolve N+1 problems in relational databases.

Community
Advanced
openscientist-ioopenscientist-io
44

kbase-query

Query and explore KBase BERDL Datalake databases and tables via MCP REST.

Official
Intermediate