ffi-capsule-protocol

Enforces the settled PyCapsule FFI protocol for DataFusion extension getters and codecs.

599|167|Updated Jul 20, 2022
One-click install
npx skills add https://github.com/apache/datafusion-python --skill ffi-capsule-protocol
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: ffi-capsule-protocol
Source: https://github.com/apache/datafusion-python/tree/main/.ai/skills/ffi-capsule-protocol
Command: npx skills add https://github.com/apache/datafusion-python --skill ffi-capsule-protocol

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

DataFusion-Python shares Rust objects with extension libraries through PyCapsules via a family of __datafusion_*__ dunder getters, and deviating from the established convention causes dangling weak references, invisible UDF registrations, and breaking API changes across extension libraries.

Core Features & Use Cases

  • Protocol Conventions: Defines the seven rules governing __datafusion_*__ capsule getters, including session-passing signatures and the ban on constructing SessionContext inside extension libraries.
  • Helper Locations: Points to the capsule extraction helpers in crates/util/src/lib.rs such as ffi_logical_codec_from_pycapsule and ffi_task_context_provider_from_pycapsule.
  • Breaking Change Procedure: Specifies the upgrade-guide entries, api change labels, and Protocol type-hint updates required when a getter signature changes.
  • Use Case: When adding a new FFI table provider getter, run the grep enumeration first, match the existing getter signature shape, and add a helper in crates/util/src/lib.rs rather than hand-rolling capsule extraction.

Quick Start

Review the FFI capsule protocol rules before I add a new __datafusion_table_provider__ getter to the extension example.

Frequently Asked Questions about ffi-capsule-protocol

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I add a new __datafusion_*__ capsule getter in datafusion-python?

First enumerate existing getters with grep over crates/ and examples to match the established signature shape. The getter takes the session it is installed on, returns a PyCapsule, and needs a corresponding extraction helper added in crates/util/src/lib.rs.

Why does my FFI codec fail with TaskContextProvider went out of scope?

FFI_TaskContextProvider holds its provider weakly, so a SessionContext constructed inline in the getter is dropped before the capsule is used. Take the provider from the passed-in session via ffi_task_context_provider_from_pycapsule instead of building a new context.

Can an extension library construct its own SessionContext for FFI codecs?

No. A fresh SessionContext is the wrong registry, so UDFs the host registered are invisible during decode, and its weak handle dangles. Use the *_with_ffi_codec constructors with codecs extracted from the host session.

Is changing a __datafusion_*__ getter signature a breaking change?

Yes. Extension libraries implement these methods, so a signature change breaks them with a bare TypeError. Add an upgrade-guide entry, apply the api change label, reuse call_capsule_getter for diagnostics, and update the Protocol hints in context.py and user_defined.py.

Why must SessionState be mutated in place instead of replacing SessionContext?

FFI codecs carry weak handles bound to one Arc<SessionContext> allocation. Replacing the allocation orphans every registered FFI provider and codec, so mutate state via state_ref().write() and preserve the session id across the rewrite.