skills.rest
OpenSourcePharmaFoundation
OpenSourcePharmaFoundationCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

data-scraper

Build and maintain scrapers that collect biomedical data into Neo4j-compatible CSV/JSON.

4|1|Updated Jan 8, 2024
Search Tags:#rate limiting#data pipeline#api integration#scraping#neo4j#csv generation#biomedical data
One-click install
npx skills add https://github.com/OpenSourcePharmaFoundation/ospf-ayurveda-kg --skill data-scraper-opensourcepharmafoundation
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-scraper
Source: https://github.com/OpenSourcePharmaFoundation/ospf-ayurveda-kg/tree/main/.claude/skills/data-scraper
Command: npx skills add https://github.com/OpenSourcePharmaFoundation/ospf-ayurveda-kg --skill data-scraper-opensourcepharmafoundation

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill removes the manual effort of assembling up-to-date biomedical data by automating data collection from public biomedical databases for the OSPF Ayurveda knowledge graph.

Core Features & Use Cases

  • Build and run scrapers: Create or update scraper modules under src/scrapers/ with consistent CLI behavior, test mode, and resumable runs.
  • Generate Neo4j-ready datasets: Transform raw API responses into clean CSV outputs tailored for LOAD CSV imports into Neo4j.
  • Integrate new data sources: Add high-priority sources (e.g., TTD, DrugBank, UniProt, KEGG, ClinicalTrials.gov, Open Targets) while respecting usage policies, rate limits, and licensing constraints.
  • Data integrity and provenance: Preserve immutable raw responses under data/raw/, produce consistent columns for identifiers and relationships, and document source provenance.

Quick Start

Run the data scraper in test mode to validate a new or modified source produces the expected processed CSV outputs for Neo4j loading.

Frequently Asked Questions about data-scraper

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I collect biomedical data from public databases for a Neo4j knowledge graph?▼

You collect biomedical data by building scraper modules that fetch drug, compound, and target information, then transforming raw API responses into Neo4j-compatible CSV files for direct LOAD CSV imports.

How do I structure scraped API data into CSV files for Neo4j LOAD CSV imports?▼

Structuring scraped API data for Neo4j involves transforming raw responses into clean CSV outputs with consistent identifier columns, using provided CSV escaping utilities to ensure compatibility with LOAD CSV conventions.

Can I use this scraper to integrate data from sources like UniProt and ClinicalTrials.gov?▼

Yes, you can integrate data from sources like UniProt and ClinicalTrials.gov by adding new scraper modules that respect each source's usage policies, rate limits, and licensing constraints during data collection.

What is the best way to handle rate limiting and licensing when scraping biomedical data?▼

The best way to handle rate limiting and licensing is configuring scraper modules to respect API usage policies during collection, preserving immutable raw responses separately in a data directory for provenance.

How do I debug scraper failures or test new biomedical data sources safely?▼

You debug scraper failures and test new data sources by running scraper modules in test mode, which validates that modified sources produce expected processed CSV outputs without risking full data collection runs.

Why does my data pipeline need to preserve raw outputs in a separate directory?▼

Your data pipeline preserves raw outputs in a separate directory to maintain data integrity and provenance, keeping immutable original API responses safe while processed CSV files are generated for graph loading.

Related Skills

Explore More Software Engineering Skills→
FindMalekFindMalek
9

prisma-database-setup

Configure Prisma for multiple database providers and set up the client.

Community
Intermediate
neo4j-contribneo4j-contrib
101

neo4j-import-skill

Import structured CSV, JSON, and bulk datasets into Neo4j.

Official
Advanced
NinatuziNinatuzi

brain-ops

Automate read and write operations for people and company entities in knowledge graphs.

Community
Intermediate
Casper-StudiosCasper-Studios
11

apify-scrapers

Automate public data extraction from social media and websites via Apify API.

Official
Advanced