skills.rest
ivanshamaev
ivanshamaevCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

starrocks-files-ingestion

Ingest and query S3/HDFS files with StarRocks FILES() and Iceberg catalogs.

14|1|Updated May 5, 2026
Search Tags:#partition pruning#starrocks#files()#iceberg catalog#s3 ingestion#hdfs csv#etl patterns
One-click install
npx skills add https://github.com/ivanshamaev/de-agent-skills --skill starrocks-files-ingestion
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: starrocks-files-ingestion
Source: https://github.com/ivanshamaev/de-agent-skills/tree/main/group_skills/starrocks_group_skills/starrocks_files_ingestion
Command: npx skills add https://github.com/ivanshamaev/de-agent-skills --skill starrocks-files-ingestion

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill solves the challenge of loading and querying data stored in external files (S3/HDFS) without first creating and managing explicit tables, while also enabling efficient integration with Iceberg external catalogs in StarRocks.

Core Features & Use Cases

  • FILEs() on-demand querying: Read Parquet/ORC/CSV directly from paths (S3/HDFS) for quick exploration or lightweight ingestion.
  • External Iceberg catalog workflows: Create Iceberg external catalogs (Hive Metastore, AWS Glue, REST, MinIO+HMS), query them, and move data into StarRocks internal tables.
  • Partition-aware performance: Use partition filter pushdown patterns with EXPLAIN to avoid scanning unnecessary partitions and to speed up ETL.
  • Schema and DDL patterns: Handle schema evolution and use safe DDL patterns like SHOW CREATE CATALOG / SHOW CREATE TABLE after CTAS.

Quick Start

Use the starrocks-files-ingestion skill to query Parquet files on S3 with an inferred schema using StarRocks FILES() without creating a table first.

Frequently Asked Questions about starrocks-files-ingestion

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I query Parquet files directly from S3 in StarRocks without creating a table first?▼

Yes, querying Parquet files directly from S3 without creating a table is possible in StarRocks using the FILES() table function. It infers schema automatically, enabling on-demand data exploration and lightweight ingestion by passing path and format parameters.

How do I create an Iceberg external catalog in StarRocks to query lakehouse data?▼

You create an Iceberg external catalog in StarRocks using the CREATE EXTERNAL CATALOG DDL. It supports Hive Metastore, AWS Glue, REST, and MinIO+HMS configurations to query lakehouse data and move it into StarRocks internal tables.

How do I use partition pruning with EXPLAIN to speed up ETL in StarRocks external catalogs?▼

You speed up ETL in StarRocks external catalogs by applying partition filter pushdown patterns and validating them using EXPLAIN. This ensures partition pruning avoids scanning unnecessary partitions during your lakehouse data movement.

Can I ingest CSV files from HDFS into StarRocks using FILES()?▼

Yes, ingesting CSV files from HDFS into StarRocks is possible using the FILES() table function. It requires appropriate S3/HDFS connectivity properties and path/format parameters to read files for one-off backfills and data exploration.

What is the best way to handle schema evolution after using CTAS with Iceberg catalogs in StarRocks?▼

The best way to handle schema evolution after using CTAS with Iceberg catalogs in StarRocks is using safe DDL patterns. You can verify table structures and schemas using SHOW CREATE CATALOG and SHOW CREATE TABLE statements.

Related Skills

Explore More Data & Analytics Skills→
ivanshamaevivanshamaev
14

starrocks-broker-load

Automate bulk data loading into StarRocks via Broker Load from cloud storage.

Community
Intermediate
ivanshamaevivanshamaev
14

starrocks-lakehouse-integration

Configure StarRocks external catalogs to query Iceberg, Hive, and Delta tables.

Community
Advanced
ivanshamaevivanshamaev
14

trino-lakehouse-platform-architect

Design Trino lakehouse architectures with Iceberg catalogs and Bronze/Silver/Gold modeling.

Community
Advanced
ivanshamaevivanshamaev
14

starrocks-materialized-views

Create and manage StarRocks materialized views with refresh and rewrite tuning.

Community
Advanced