skills.rest
mdoucet
mdoucetCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-packaging

Package time-resolved neutron reflectometry data into Parquet datasets for Apache Iceberg.

Updated Aug 27, 2026
Search Tags:#parquet#iceberg#data-lake#tnr#data-packaging#eis#reflectometry
One-click install
npx skills add https://github.com/mdoucet/analyzer --skill data-packaging
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-packaging
Source: https://github.com/mdoucet/analyzer/tree/main/skills/data-packaging
Command: npx skills add https://github.com/mdoucet/analyzer --skill data-packaging

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Package reduced time-resolved neutron reflectometry (tNR) outputs together with EIS timing metadata into a standardized Parquet dataset so data can be ingested into an Apache Iceberg lakehouse and used directly for downstream analysis.

Core Features & Use Cases

  • Combine reduced reflectivity and EIS metadata: Merges per-run reflectivity files with interval JSON and the reduction template into a single analysis-ready Parquet table.
  • Produce both data and metadata artifacts: Emits a main tnr_data.parquet with per-file reflectivity arrays and a tnr_data_metadata.parquet containing experiment-level metadata, reduction template, and full intervals JSON.
  • Validation and safe packaging: Supports input validation mode to check required files and formats without creating outputs, and documents column mappings and Q/R ranges for reproducibility.
  • Use Case: Prepare a full experiment's reduced tNR results for large-scale analytics and storage in an Iceberg lakehouse, enabling downstream model fitting, visualization, or ML workflows.

Quick Start

Package reduced tNR results using intervals.json, results/tnr, and template.xml into results/tnr/tnr_dataset.parquet.

Frequently Asked Questions about data-packaging

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I package time-resolved neutron reflectometry data for an Iceberg data lake?▼

Package time-resolved neutron reflectometry data for an Apache Iceberg data lake by merging reduced tNR reflectivity files with EIS interval JSON and a reduction template XML into analysis-ready Parquet datasets.

What is included in the Parquet output when combining tNR reflectivity and EIS metadata?▼

The Parquet output includes a main tnr_data.parquet file containing mapped reflectivity arrays and a tnr_data_metadata.parquet file holding experiment-level metadata, reduction templates, and full EIS intervals JSON.

How do I validate reduced tNR reflectivity files and EIS intervals before generating Parquet?▼

Validate reduced tNR reflectivity files and EIS intervals by running the packaging process in validate-only mode, which checks required input files and formats without creating the final Parquet outputs.

Can I use Apache Iceberg lakehouse storage for downstream analytics on tNR reflectivity datasets?▼

Apache Iceberg lakehouse storage supports downstream analytics on tNR reflectivity datasets by directly ingesting the standardized Parquet outputs for model fitting, visualization, or ML workflows.

What input files are required to prepare tNR reflectivity data for Parquet ingestion?▼

Preparing tNR reflectivity data for Parquet ingestion requires reduced tNR reflectivity files, an EIS interval JSON file containing timing metadata, and a reduction template XML for data lake preparation.

Related Skills

Explore More Data & Analytics Skills→
talmolabtalmolab
1

lab-roster

Generate lab member rosters in Markdown and CSV from member files and git history.

Official
Intermediate
WbunkerWbunker

apache-iceberg-expert

Explain Apache Iceberg architecture, metadata layers, and catalog management.

Community
Advanced
sandraschisandraschi
18

chakra-and-energy-systems-expert

Explain chakra systems, energy healing, and aura interpretation for educational contexts.

Community
Basic
JaviMontanoJaviMontano

metodologia-data-engineering

Automate data platform architecture design for ingestion, storage, quality, and observability.

Community
Advanced