skills.rest
dchuk
dchukCOMMUNITY
Explore all skills by this author →
GitHub

sm-scraper-adapter

Create and register custom scraper adapters inheriting from SourceMonitor::Scrapers::Base.

3|Updated Oct 16, 2025
Search Tags:#http#scraper#ruby#rails#adapter#web-scraping#content-extraction
One-click install
npx skills add https://github.com/dchuk/source_monitor --skill sm-scraper-adapter
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: sm-scraper-adapter
Source: https://github.com/dchuk/source_monitor/tree/main/.claude/skills/sm-scraper-adapter
Command: npx skills add https://github.com/dchuk/source_monitor --skill sm-scraper-adapter

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Designing scalable data-scraping components for SourceMonitor by enabling custom adapters that integrate into the scraping pipeline with a consistent contract.

Core Features & Use Cases

  • Inherits from SourceMonitor::Scrapers::Base and implements the #call method to return a well-defined Result including status, html, content, and metadata.
  • Supports adapter registration and per-source assignment via source.scrape_settings, enabling flexible deployments across multiple feeds.
  • Includes a clear path for extending with built-in adapters and custom ones, with a defined adapter contract and error handling.

Quick Start

Create your adapter by defining a class that inherits from SourceMonitor::Scrapers::Base, implement #call, register the adapter, and assign it to a source using source.scrape_settings.

Frequently Asked Questions about sm-scraper-adapter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I create a custom web scraper adapter in Ruby for content extraction?▼

To create a custom web scraper adapter, inherit from the base scraper class and implement a #call method that returns a structured Result with status, html, content, and metadata. Register the adapter in the initialization flow so each source can select the appropriate scraping class.

How do I configure per-source scrape settings for a Ruby web scraper?▼

Per-source scraping settings are configured by deep-merging adapter defaults, source overrides, and invocation overrides via source.scrape_settings. This allows flexible deployments across multiple feeds while maintaining a consistent adapter contract.

How does error handling work for Rails web scraping adapters?▼

When scraping failures occur, the web scraping adapter produces a meaningful metadata payload within the Result object. This ensures the pipeline receives structured error information rather than crashing the data extraction process.

What is a scraper adapter contract for HTTP content extraction?▼

A scraper adapter contract requires inheriting from the base scraper class and implementing a #call method. This method must return a well-defined Result object containing status, html, content, and metadata to integrate into the scraping pipeline.

How do I register multiple scraping adapters for different web sources?▼

Register custom adapters in the initialization flow and assign them to sources using source.scrape_settings. This registration mechanism lets each source dynamically select the appropriate scraping class during pipeline execution.

Can I override default adapter settings for a specific web scraping invocation?▼

Yes, web scraping adapter settings support deep-merging from adapter defaults, source overrides, and invocation overrides. This layered configuration allows you to customize scraping behavior per source or per individual execution.

Related Skills

Explore More Software Engineering Skills→
whmathews15whmathews15
1

andrew-kane-gem-writer

Design production-ready Ruby gems with Minitest and lazy Rails loading.

Community
Advanced
ComwrapUkReplyComwrapUkReply

scrape-webpage

Scrape webpages to extract content, metadata, and images for migration workflows.

Community
Advanced
luisgustavooliveiraluisgustavooliveira

scrape

Extract text and structured data from web pages using Python libraries.

Community
Intermediate
LuminairPrimeLuminairPrime

mcp-firecrawl

Scrape web pages and extract structured content using Firecrawl.

Community
Advanced