research

Perform multi-engine web searches and structured OSINT investigations via HTTP endpoints.

1|Updated Mar 18, 2024
One-click install
npx skills add https://github.com/erfianugrah/dotfiles --skill research-erfianugrah
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: research
Source: https://github.com/erfianugrah/dotfiles/tree/main/.pi/agent/skills/research
Command: npx skills add https://github.com/erfianugrah/dotfiles --skill research-erfianugrah

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

It eliminates slow, manual OSINT workflows by providing a single set of HTTP endpoints for multi-engine search, clean content extraction, and structured investigation across domains, IPs, emails, usernames, URLs, phones, and reputations.

Core Features & Use Cases

  • Multi-engine web search (SearXNG): Aggregate results across 7+ search engines to quickly discover relevant sources and leads for investigation or research.
  • Clean-content fetching for SPAs and webpages: Convert web pages into boilerplate-stripped Markdown using a fast extraction path with a Playwright fallback for JavaScript-heavy sites.
  • OSINT investigation across multiple intelligence types: Perform DNS/subdomain discovery, IP geo/ports/CVE correlations, email registration and breach checks (when configured), username scans across many platforms, urlscan.io lookups/submissions, phone parsing/validation, VirusTotal threat reputation, and NVD CVE lookups, with optional long-running job polling.

Quick Start

Use the OSINT stack to investigate a domain by sending an authenticated request to the local OSINT service: run curl with a POST to /investigate/domain using JSON with the target domain and desired mode.

Frequently Asked Questions about research

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate domain investigation and OSINT lookups across multiple platforms?

Domain investigation and OSINT lookups are automated by sending authenticated HTTP POST requests to an OSINT service, which coordinates DNS discovery, IP correlation, and username scans across many platforms. You use JSON payloads to specify the target and desired mode.

Can I extract clean text content from JavaScript-heavy single page applications?

Yes, clean content extraction from JavaScript-heavy single page applications is handled by a crawler extractor service that strips boilerplate into Markdown. It uses a fast extraction path first and falls back to Playwright to render dynamic content.

What services do I need to run locally for multi-engine web search and OSINT investigations?

You need SearXNG running on port 8888 for multi-engine search, a crawler extractor on port 8889 for webpage fetching, and an OSINT service on port 8890 for structured investigations. All endpoints require bearer authentication.

Does threat intelligence gathering support long-running asynchronous jobs?

Threat intelligence gathering supports long-running asynchronous jobs by submitting targets to endpoints like urlscan.io lookups and polling the /jobs/{id} path. This allows you to retrieve structured results once the background investigation completes.

How do I look up CVEs and check IP reputation during security research?

CVE lookups and IP reputation checks are performed by querying the OSINT service's /investigate/* endpoints, which correlate IP ports with NVD CVE data and fetch VirusTotal threat reputation scores. You submit the target indicator via JSON in an authenticated request.

What is the best way to aggregate web search results from 7+ search engines?

The best way to aggregate web search results is to query a local SearXNG instance on port 8888, which merges results across 7+ search engines. This provides a single endpoint to discover relevant sources and leads for research.