unf

Name: unf
Availability: InStock
Author: CODATA

Generate order-invariant data fingerprints for integrity verification using Python and Polars.

8|1|Updated Mar 14, 2026

Search Tags:#data integrity #reproducibility #hash #format invariance #dataset verification

Ready to run

npx skills add https://github.com/codata/croissant-toolkit --skill unf

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: unf
Source: https://github.com/codata/croissant-toolkit/tree/main/.gemini/skills/unf
Command: npx skills add https://github.com/codata/croissant-toolkit --skill unf

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires polars, dartfx-unf, and includes scripts (resource) components.

What problem does it solve?

This Skill provides a reliable mechanism for generating unique, format-invariant fingerprints for data objects, ensuring data integrity and facilitating reproducible analyses.

Core Features & Use Cases

Semantic Hashing: Creates order-invariant fingerprints for strings and data columns, enabling consistent identification across formats.
Vector and File Hashing: Supports dataframes and raw data files in CSV, Parquet, SAS, and Stata formats to produce reproducible identifiers.
Use Case: Verify that datasets stored in different formats or with columns reordered produce the same identifier for version control and data validation.

Quick Start

Use the unf skill to generate a fingerprint for your dataset stored in "data.parquet" to verify data consistency across formats.

Frequently Asked Questions about unf

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I generate a consistent hash for a dataset stored in different formats like CSV and Parquet?▼

A format-invariant data fingerprint provides a unique, order-invariant hash for your dataset. It works by creating semantic hashes for strings and data columns, ensuring that reordering columns or changing file formats does not alter the final dataset verification identifier.

Can I verify data integrity across large datasets using Polars?▼

Yes, you can verify data integrity across large datasets using Polars. The implementation requires Python and Polars to process large dataframes, applying order-invariant hashing to generate reproducible fingerprints for robust dataset validation and version control.

What is the best way to check if two dataframes have identical content despite column reordering?▼

The best way to check if two dataframes have identical content despite column reordering is semantic hashing. It generates order-invariant fingerprints for data columns, ensuring restructured or reordered data yields the exact same reproducible hash for consistent dataset identification.

Do I need the dartfx-unf library to generate reproducible data fingerprints?▼

Yes, you need the dartfx-unf library along with Python and Polars to generate reproducible data fingerprints. These dependencies are required to apply format-agnostic hashing and ensure high-assurance data integrity verification across large datasets.

Why does my dataset fingerprint change when I save a file as CSV instead of Parquet?▼

Your dataset fingerprint changes because standard hashing algorithms are format-dependent. Using format-invariant hashing solves this by creating semantic fingerprints based on the actual data content, ensuring CSV and Parquet files with identical data produce matching reproducible identifiers.

When do I need format-invariant hashing for data management workflows?▼

You need format-invariant hashing for data management workflows when verifying data integrity and facilitating reproducible analyses. It is essential when comparing datasets stored in different formats or with reordered columns for version control and data validation.