skills.rest
etylsarin
etylsarinCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-engineering

Build ETL and web scraping pipelines that extract and normalize data to NDJSON.

59|12|Updated Feb 26, 2026
Search Tags:#etl#scraping#pipelines#cms#web-scraping#data-engineering#ndjson
One-click install
npx skills add https://github.com/etylsarin/opencastle --skill data-engineering-etylsarin
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/etylsarin/opencastle/tree/main/src/orchestrator/skills/data-engineering
Command: npx skills add https://github.com/etylsarin/opencastle --skill data-engineering-etylsarin

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill provides patterns and operational guidance to build reliable data pipelines that extract, normalize, and import large-scale web and CMS data, reducing manual scraping and messy ad-hoc imports.

Core Features & Use Cases

  • Scraping & ETL workflows: Browser-based scraping clusters, anti-detection strategies, retry and checkpointing, and composable pipeline stages for robust extraction.
  • NDJSON output & CMS import: Line-delimited JSON output conventions, deterministic IDs, idempotent createOrReplace imports, and streaming processing for large files.
  • Production readiness: Error recovery, dry-run modes, backup-before-bulk, structured address normalization, and data lineage tracking for safe bulk operations.

Quick Start

Use the data-engineering skill to scrape public site data, normalize records to NDJSON, and prepare an idempotent import into your CMS.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build an ETL pipeline for web scraping and CMS imports?▼

Build ETL pipelines by compositing scraping stages with retry, checkpointing, and NDJSON normalization to extract data and prepare idempotent createOrReplace imports into your CMS.

What is the best way to process large NDJSON files for data ingestion?▼

Process large NDJSON files using stream-based processing to handle large-scale web and CMS data, applying deterministic IDs and schema conventions for reliable ingestion.

How does checkpoint recovery work in a web scraping pipeline?▼

Checkpoint recovery in scraping pipelines captures progress during extraction, enabling retry logic and error recovery to resume data ingestion without duplicating previously processed records.

Can I use this for compliant public web scraping with rate limiting?▼

Yes, you can scrape public sites using respectful scraping practices, applying rate limiting and robots.txt compliance to manage browser-based extraction clusters safely.

Does this approach support dry-run modes and backups for bulk CMS imports?▼

Yes, bulk CMS imports support dry-run modes and backup-before-bulk operations, ensuring safe data ingestion with structured address normalization and data lineage tracking.

Why do I need NDJSON output conventions for data engineering pipelines?▼

NDJSON output conventions provide line-delimited JSON formatting with deterministic IDs, enabling streaming processing and idempotent imports required for reliable large-scale data ingestion.

Related Skills

Explore More Data & Analytics Skills→
while-basicwhile-basic

enact/firecrawl

Extract web page content as Markdown or structured data via the Firecrawl API.

Community
Advanced
GDSDNGDSDN
3

db-load-csv

Import CSV data into PostgreSQL using a staging table and validation.

Community
Advanced
simotasimota
68

Stream

Design batch, streaming, and hybrid data pipelines with quality gates and recovery paths.

Community
Advanced
RanaPriyanshRanaPriyansh

twitter-scraping

Scrape Twitter profiles, followers, tweets, and media via Node.js or browser scripts.

Community
Intermediate