skills.rest
awslabs
awslabsOFFICIAL
Explore all skills by this author →
WebsiteGitHub

dataset-evaluation

Validates JSONL dataset formatting and schema compliance for SageMaker model fine-tuning and evaluation.

881|152|Updated Feb 5, 2026
Search Tags:#fine-tuning#S3#JSONL#format detection#dataset validation#Sagemaker#Machine learning
One-click install
npx skills add https://github.com/awslabs/agent-plugins --skill dataset-evaluation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: dataset-evaluation
Source: https://github.com/awslabs/agent-plugins/tree/main/plugins/sagemaker-ai/skills/dataset-evaluation
Command: npx skills add https://github.com/awslabs/agent-plugins --skill dataset-evaluation

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires boto3, and includes scripts (resource) and references (resource) components.

What problem does it solve?

Before launching a SageMaker fine-tuning job, you need certainty that your training or evaluation dataset matches the exact schema required by your chosen model family (Nova, GPT-OSS, open weights) and technique (SFT, DPO, RLVR). This Skill detects the file format, checks schema compliance, and reports whether the data is ready for training or evaluation.

Core Features & Use Cases

  • Automatic Format Detection: Samples the first 1MB of a local or S3 JSONL file and classifies it across 11 supported formats including Nova SFT/DPO/RLVR, GPT-OSS, Open Weights, Verl, and SageMaker Eval.
  • Schema Validation: Checks required fields, message structures, and role/content types per record, reporting line-numbered errors with confidence levels.
  • Strategy-Aware Guidance: Compares the detected format against the selected fine-tuning strategy and model, warns about mismatches, and recommends dataset transformation when needed.
  • Use Case: You have a JSONL file in S3 and want to fine-tune a Nova model with DPO. Run this Skill to confirm the dataset uses the candidates/preferenceLabel structure before starting the training job.

Quick Start

Ask the assistant to check whether your dataset file at a given local path or S3 URI is valid for your planned SageMaker fine-tuning job.

Frequently Asked Questions about dataset-evaluation

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I validate a JSONL dataset for SageMaker fine-tuning?▼

Run the format_detector.py script with your local file path or S3 URI as the argument. It samples the first 1MB, detects the format among 11 supported types, and reports whether the data is valid with line-numbered errors.

What dataset formats does SageMaker fine-tuning support?▼

Supported formats include Nova SFT, Nova DPO, Nova RLVR, GPT-OSS SFT and DPO, Open Weights SFT and DPO, Verl, Verl Legacy, and SageMaker Eval. Each has distinct schema requirements such as prompt/completion pairs or messages with candidates.

Can I validate a dataset stored in S3 without downloading it?▼

Yes, the format detector reads S3 files directly using a Range request to sample the first 1MB. It does not download the full file or make local copies of your data.

What is the difference between training and evaluation dataset formats?▼

Training datasets must match the fine-tuning strategy format such as SFT prompt/completion or DPO chosen/rejected pairs. Evaluation datasets use a query/response structure, with scorer-specific requirements for Prime Math, Prime Code, or Custom Lambda evaluation.

Why does my dataset fail validation for my chosen model?▼

Failures occur when the detected format does not match the selected model family and strategy, such as using flat string content for Nova models that require nested content arrays. The validator reports the expected versus detected format per line.

Related Skills

Explore More Software Engineering Skills→
webdevtodayjasonwebdevtodayjason

axolotl

Fine-tune large language models with Axolotl using YAML configurations.

Community
Advanced
wangx7wangx7

agent-platform-tuning

Fine-tune open-source and Gemini models on Agent Platform infrastructure.

Community
Advanced
surus-latsurus-lat
8

whisper-benchmark

Benchmark Whisper-family ASR models on FLEURS datasets with WER and CER metrics.

Official
Advanced
rjonhaasrjonhaas
1

Skill: Hypothesis Engine (Working-Theory Ledger)

Maintain an append-only JSONL ledger scoring investigative hypotheses against evolving evidence.

Community
Advanced