skills.rest
apache
apacheOFFICIAL
Explore all skills by this author →
WebsiteGitHub

datafusion-python

Write DataFusion DataFrame and SQL queries with Apache Arrow integration in Python.

595|165|Updated Jul 20, 2022
Search Tags:#python#sql#interoperability#dataframe#datafusion#expressions#sessioncontext
One-click install
npx skills add https://github.com/apache/datafusion-python --skill datafusion-python
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: datafusion-python
Source: https://github.com/apache/datafusion-python/tree/main/skills/datafusion_python
Command: npx skills add https://github.com/apache/datafusion-python --skill datafusion-python

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

DataFusion Python bindings enable Python developers to run DataFusion queries via the DataFrame API and SQL, bridging Python workloads with an in-process query engine built on Apache Arrow for fast analytics.

Core Features & Use Cases

  • Data loading from Parquet, CSV, and JSON sources into a SessionContext.
  • DataFrame-based query construction with lazy evaluation and a rich expression API (Expr, col, lit, functions as F).
  • SQL-to-DataFrame mappings and interoperability with pandas, polars, and other in-memory data frames.
  • Idiomatic patterns and common pitfalls guidance for practical analytics tasks.

Quick Start

Create a SessionContext, load data, and run a SQL or DataFrame query.

Frequently Asked Questions about datafusion-python

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run SQL queries on Parquet and CSV files in Python?▼

Run SQL queries on Parquet and CSV files in Python by loading them into a DataFusion SessionContext. This bridges Python workloads with an in-process Apache Arrow query engine for fast, in-memory analytics directly on the loaded data.

What is the best way to map SQL queries to DataFrame operations in Python?▼

Map SQL to DataFrame operations in Python using DataFusion bindings. You construct queries using a lazy DataFrame API with a rich expression API, enabling seamless translation between SQL statements and programmatic DataFrame transformations on Apache Arrow data.

Can I use DataFusion DataFrames with pandas and Polars?▼

Yes, DataFusion DataFrames interoperate with pandas, Polars, cuDF, and PyArrow. The bindings map SQL and DataFrame operations to these in-memory formats, allowing integration of DataFusion query results into existing Python analytics workflows.

How do I build data processing workflows with the DataFusion Python expression API?▼

Build data processing workflows in Python by creating a SessionContext, loading data, and applying transformations using the DataFusion expression API. Use functions, column references, and literals to construct lazy DataFrame operations for analytics tasks.

What are common pitfalls when using DataFusion Python bindings for dataframe operations?▼

Common pitfalls when using DataFusion Python bindings for dataframe operations typically involve misunderstanding lazy evaluation behavior and expression building syntax. The bindings provide idiomatic patterns and guidance to help avoid these issues during practical analytics tasks.

Related Skills

Explore More Data & Analytics Skills→
spqwspqw

observable-framework-sql

Execute SQL queries on CSV, JSON, Arrow, and Parquet files with DuckDB in Observable Framework.

Community
Intermediate
Umbral-BotUmbral-Bot

big-data-python

Process large datasets with Pandas, Polars, and Dask.

Official
Advanced
legoutlegout
3

duckdb

Execute SQL analytical queries on local files and DataFrames.

Community
Intermediate
silvainfmsilvainfm
2

duckdb

Run SQL analytics directly on CSV, Parquet, and JSON files and Pandas or Polars DataFrames.

Community
Advanced