sm-scraper-adapter

Name: sm-scraper-adapter
Availability: InStock
Author: dchuk

Create and register custom scraper adapters inheriting from SourceMonitor::Scrapers::Base.

3|Updated Oct 16, 2025

Search Tags:#http #scraper #ruby #rails #adapter #web-scraping #content-extraction

Ready to run

npx skills add https://github.com/dchuk/source_monitor --skill sm-scraper-adapter

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: sm-scraper-adapter
Source: https://github.com/dchuk/source_monitor/tree/main/.claude/skills/sm-scraper-adapter
Command: npx skills add https://github.com/dchuk/source_monitor --skill sm-scraper-adapter

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Designing scalable data-scraping components for SourceMonitor by enabling custom adapters that integrate into the scraping pipeline with a consistent contract.

Core Features & Use Cases

Inherits from SourceMonitor::Scrapers::Base and implements the #call method to return a well-defined Result including status, html, content, and metadata.
Supports adapter registration and per-source assignment via source.scrape_settings, enabling flexible deployments across multiple feeds.
Includes a clear path for extending with built-in adapters and custom ones, with a defined adapter contract and error handling.

Quick Start

Create your adapter by defining a class that inherits from SourceMonitor::Scrapers::Base, implement #call, register the adapter, and assign it to a source using source.scrape_settings.

Frequently Asked Questions about sm-scraper-adapter

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I create a custom web scraper adapter in Ruby for content extraction?▼

To create a custom web scraper adapter, inherit from the base scraper class and implement a #call method that returns a structured Result with status, html, content, and metadata. Register the adapter in the initialization flow so each source can select the appropriate scraping class.

How do I configure per-source scrape settings for a Ruby web scraper?▼

Per-source scraping settings are configured by deep-merging adapter defaults, source overrides, and invocation overrides via source.scrape_settings. This allows flexible deployments across multiple feeds while maintaining a consistent adapter contract.

How does error handling work for Rails web scraping adapters?▼

When scraping failures occur, the web scraping adapter produces a meaningful metadata payload within the Result object. This ensures the pipeline receives structured error information rather than crashing the data extraction process.

What is a scraper adapter contract for HTTP content extraction?▼

A scraper adapter contract requires inheriting from the base scraper class and implementing a #call method. This method must return a well-defined Result object containing status, html, content, and metadata to integrate into the scraping pipeline.

How do I register multiple scraping adapters for different web sources?▼

Register custom adapters in the initialization flow and assign them to sources using source.scrape_settings. This registration mechanism lets each source dynamically select the appropriate scraping class during pipeline execution.

Can I override default adapter settings for a specific web scraping invocation?▼

Yes, web scraping adapter settings support deep-merging from adapter defaults, source overrides, and invocation overrides. This layered configuration allows you to customize scraping behavior per source or per individual execution.