skills.rest
mims-harvard
mims-harvardOFFICIAL
Explore all skills by this author →
Website𝕏@marinkazitnikGitHub

tooluniverse-data-wrangling

Download and parse scientific data from APIs and file formats using Python code.

1.7k|254|Updated Mar 3, 2025
Search Tags:#bioinformatics#scientific data#Pandas#data wrangling#file parsing#bulk download#api download
One-click install
npx skills add https://github.com/mims-harvard/ToolUniverse --skill tooluniverse-data-wrangling
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tooluniverse-data-wrangling
Source: https://github.com/mims-harvard/ToolUniverse/tree/main/plugins/tooluniverse/skills/tooluniverse-data-wrangling
Command: npx skills add https://github.com/mims-harvard/ToolUniverse --skill tooluniverse-data-wrangling

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Scientific data often lives outside what dedicated tools return: tools may only give metadata, truncate results to 10-100 records, or not exist for a source at all. This Skill provides universal patterns for downloading bulk records and parsing specialized formats (VCF, h5ad, BAM, SDF, GCT, mzML, NIfTI) directly with Python.

Core Features & Use Cases

  • Format Cookbook: Ready-to-use Python snippets for parsing tabular, genomics, structural, omics, mass spectrometry, neuroimaging, phylogenetics, and compressed file formats.
  • API Patterns by Domain: Direct REST API workflows for 24 domains including NCBI, EBI, GDC, CDC, GWAS Catalog, PubChem, ClinicalTrials.gov, and EuropePMC, with pagination, rate limiting, and retry logic.
  • Restricted Source Guidance: Access requirements and programmatic download paths for credentialed sources like UK Biobank, dbGaP, MIMIC-IV, and ADNI.
  • Use Case: You need all GWAS associations for diabetes. Instead of paginating a tool's limited results, download the full GWAS Catalog TSV and filter locally with pandas.

Quick Start

Ask the agent to download all ClinVar variants for a gene and parse them into a pandas DataFrame using the data wrangling patterns.

Frequently Asked Questions about tooluniverse-data-wrangling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I download bulk data from NCBI or UniProt APIs?▼

Use the E-utilities or UniProt REST pattern: search to collect IDs, then fetch records in batches of 500. For UniProt, follow cursor pagination via the Link header until no next page remains.

How to parse VCF, BAM, or h5ad files in Python?▼

Parse VCF by stripping header lines and loading with pandas read_csv. BAM files require pysam's AlignmentFile for region queries. h5ad single-cell files load with anndata.read_h5ad.

When should I write Python code instead of using a ToolUniverse tool?▼

Write code when you need bulk downloads, custom filtering, format conversion, or thousands of records. Use tools for single record lookups and searches returning under 100 results.

How do I handle API rate limits and retries when downloading data?▼

Implement a retry loop that checks for HTTP 429 responses and waits per the Retry-After header, with exponential backoff for other failures. Most scientific APIs like NCBI, EBI, and PubChem need no authentication.

Can I access restricted datasets like UK Biobank or MIMIC-IV programmatically?▼

Yes, after obtaining credentials. UK Biobank and dbGaP require institutional applications taking weeks to months, while MIMIC-IV needs PhysioNet credentialing. Once approved, data downloads as CSV, Parquet, or VCF files.

Why does my API download return an HTML error page instead of data?▼

Servers like CDC and NCBI sometimes return HTTP 200 with HTML error pages for missing files. Guard against this by checking whether the response content starts with HTML doctype markers before parsing as JSON.

Related Skills

Explore More Data & Analytics Skills→
mims-harvardmims-harvard
1.6k

tooluniverse-disease-research

Generate fully cited disease reports with evidence-graded data from 100+ tools.

Official
Advanced
Ming-Kai-LCMing-Kai-LC

data-science-educator

Create structured data science lessons covering pandas, visualization, and machine learning workflows.

Community
Basic
AppsilonAppsilon
35

sdtm-to-adam

Generate ADaM datasets from SDTM sources with R scripts and JSON outputs.

Official
Advanced
runningZ1runningZ1
634

union-search-skill

Aggregate and normalize search results from 20+ platforms into a single JSON structure.

Community
Advanced