datafusion-python

Write DataFusion DataFrame and SQL queries with Apache Arrow integration in Python.

595|165|Updated Jul 20, 2022

Search Tags:#python #sql #interoperability #dataframe #datafusion #expressions #sessioncontext

Ready to run

npx skills add https://github.com/apache/datafusion-python --skill datafusion-python

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: datafusion-python
Source: https://github.com/apache/datafusion-python/tree/main/skills/datafusion_python
Command: npx skills add https://github.com/apache/datafusion-python --skill datafusion-python

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

DataFusion Python bindings enable Python developers to run DataFusion queries via the DataFrame API and SQL, bridging Python workloads with an in-process query engine built on Apache Arrow for fast analytics.

Core Features & Use Cases

Data loading from Parquet, CSV, and JSON sources into a SessionContext.
DataFrame-based query construction with lazy evaluation and a rich expression API (Expr, col, lit, functions as F).
SQL-to-DataFrame mappings and interoperability with pandas, polars, and other in-memory data frames.
Idiomatic patterns and common pitfalls guidance for practical analytics tasks.

Quick Start

Create a SessionContext, load data, and run a SQL or DataFrame query.

Frequently Asked Questions about datafusion-python

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I run SQL queries on Parquet and CSV files in Python?▼

Run SQL queries on Parquet and CSV files in Python by loading them into a DataFusion SessionContext. This bridges Python workloads with an in-process Apache Arrow query engine for fast, in-memory analytics directly on the loaded data.

What is the best way to map SQL queries to DataFrame operations in Python?▼

Map SQL to DataFrame operations in Python using DataFusion bindings. You construct queries using a lazy DataFrame API with a rich expression API, enabling seamless translation between SQL statements and programmatic DataFrame transformations on Apache Arrow data.

Can I use DataFusion DataFrames with pandas and Polars?▼

Yes, DataFusion DataFrames interoperate with pandas, Polars, cuDF, and PyArrow. The bindings map SQL and DataFrame operations to these in-memory formats, allowing integration of DataFusion query results into existing Python analytics workflows.

How do I build data processing workflows with the DataFusion Python expression API?▼

Build data processing workflows in Python by creating a SessionContext, loading data, and applying transformations using the DataFusion expression API. Use functions, column references, and literals to construct lazy DataFrame operations for analytics tasks.

What are common pitfalls when using DataFusion Python bindings for dataframe operations?▼

Common pitfalls when using DataFusion Python bindings for dataframe operations typically involve misunderstanding lazy evaluation behavior and expression building syntax. The bindings provide idiomatic patterns and guidance to help avoid these issues during practical analytics tasks.