skills.rest
NVIDIA
NVIDIAOFFICIAL
Explore all skills by this author →
WebsiteGitHub

udf-benchmark

Validate and compare Spark UDF implementations across CPU and GPU paths.

993|294|Updated May 14, 2020
Search Tags:#performance#validation#benchmarking#gpu#spark#cpu#udf
One-click install
npx skills add https://github.com/NVIDIA/cudf-spark --skill udf-benchmark
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: udf-benchmark
Source: https://github.com/NVIDIA/cudf-spark/tree/main/skills/udf-benchmark
Command: npx skills add https://github.com/NVIDIA/cudf-spark --skill udf-benchmark

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill helps you validate and benchmark a Spark UDF conversion by comparing the original CPU implementation against the GPU implementation, so you can confirm correctness and measure performance gains with confidence.

Core Features & Use Cases

  • Benchmark Workflow Guidance: Walks through the full benchmarking process from implementing BenchUtils to running validation and large-scale performance tests.
  • CPU vs GPU Comparison: Supports side-by-side benchmarking of CPU UDFs and RapidsUDF or SQL equivalents.
  • Microbenchmark Support: Includes optional cuDF microbenchmarks for deeper GPU-only performance analysis after Spark-level benchmarking.
  • Use Case: A developer has converted a UDF to GPU and needs to validate small-sample correctness, generate production-sized benchmark data, and compare throughput across CPU and GPU runs.

Quick Start

Use this skill to validate your BenchUtils implementation, generate benchmark data, run CPU and GPU benchmarks, and complete cuDF microbenchmarks when applicable.

Frequently Asked Questions about udf-benchmark

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I benchmark Spark UDF performance between CPU and GPU?▼

To benchmark Spark UDF performance, you implement deterministic BenchUtils configurations to generate parquet test data, validate correctness on small datasets, and run full-scale performance tests comparing CPU UDFs against GPU RapidsUDF equivalents.

What is the best way to validate a RapidsUDF conversion against the original CPU UDF?▼

Validating a RapidsUDF conversion requires running side-by-side benchmarking on small sample datasets to confirm correctness before generating production-sized parquet data and comparing throughput across full-scale CPU and GPU runs.

Can I run cuDF microbenchmarks after Spark UDF benchmarking?▼

Yes, you can run optional cuDF microbenchmarks after completing Spark-level UDF benchmarking to perform deeper GPU-only performance analysis for your RapidsUDF conversions.

Do I need executable shell scripts and parquet data generation for Spark UDF validation?▼

Yes, Spark UDF validation requires deterministic BenchUtils configuration, executable shell scripts, and parquet data generation to successfully capture and compare benchmark results across CPU and GPU execution paths.

What are the limitations of Spark UDF benchmarking with BenchUtils?▼

Spark UDF benchmarking with BenchUtils requires deterministic configurations and executable shell scripts; it applies specifically to end-to-end UDF conversion workflows and may not support non-deterministic UDF validation logic.

Related Skills

Explore More Software Engineering Skills→
NVIDIANVIDIA
993

udf-gen-test

Generate ScalaTest unit tests for Apache Spark UDFs before GPU conversion.

Official
Advanced
NVIDIANVIDIA
993

udf-optimize-cudf

Optimize cuDF RapidsUDF implementations for GPU performance with profiling and benchmarking.

Official
Advanced
NVIDIANVIDIA
993

udf-convert-to-sql

Convert Apache Spark UDFs into equivalent Spark SQL expressions.

Official
Intermediate
NVIDIANVIDIA
993

udf-convert-to-cudf

Convert Apache Spark UDFs into GPU-accelerated RapidsUDF implementations.

Official
Advanced