pulse

Scrape trending AI/tech ideas from YouTube, X, TikTok, and Anthropic changelogs into a scored, deduplicated markdown backlog.

54|7|Updated Apr 19, 2026

Search Tags:#content discovery #deduplication #scraping #transcripts #ai topics #sqlite backlog #scheduled pipeline

Ready to run

npx skills add https://github.com/robinsadeghpour/content-workflow --skill pulse-robinsadeghpour

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: pulse
Source: https://github.com/robinsadeghpour/content-workflow/tree/main/.claude/skills/pulse
Command: npx skills add https://github.com/robinsadeghpour/content-workflow --skill pulse-robinsadeghpour

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

Pulse removes the need for manual topic hunting by continuously scraping multiple sources for trending AI/tech ideas and turning them into a scored, deduplicated backlog you can use immediately.

Core Features & Use Cases

Multi-source discovery: Collects ideas from YouTube, X/Twitter, TikTok, and Anthropic changelogs, then filters them into a daily set relevant to Robin’s brand.
Scoring + deduplication: Assigns a numeric score using recency and engagement and removes duplicates via SHA-256 URL hashing against data/content.db.
Transcript enrichment + review output: Eagerly fetches YouTube and TikTok transcripts via Supadata and writes a dated review markdown file at data/review/YYYY-MM-DD.md for quick human selection.
Use case: If you wake up to a ready-to-decide backlog, Pulse ensures you have fresh content angles every morning without spending time searching across platforms.

Quick Start

Run Pulse to generate today’s topic review by executing the command to start the pipeline: run node scripts/pulse.js.

Frequently Asked Questions about pulse

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I automate scraping trending AI topics from YouTube, X, and TikTok?▼

Automate scraping trending AI topics by running a scheduled pipeline that collects content from YouTube, X/Twitter, TikTok, and Anthropic changelogs. The pipeline applies engagement scoring and deduplication to output a ranked backlog.

How does deduplication work for scraped content backlogs?▼

Deduplication works by hashing source URLs with SHA-256 and checking them against an initialized SQLite database. This prevents duplicate content ideas from entering your daily discovery backlog.

Do I need Apify and Supada credentials to fetch YouTube and TikTok transcripts?▼

Yes, you need Apify credentials for scraping sources and Supadata credentials for transcript enrichment. These services fetch video transcripts and social media data required to generate the final review.

What is the best way to generate a daily markdown review of trending tech content?▼

Generate a daily markdown review by executing a scheduled pipeline script that processes scraped sources, enriches them with transcripts, and writes a dated output file for downstream content drafting.

Can I use SQLite for storing a deduplicated content discovery backlog?▼

Yes, SQLite stores the deduplicated content discovery backlog using SHA-256 URL hashing. You must initialize the database file before running the pipeline to ensure proper tracking.

How are trending content ideas scored for relevance and recency?▼

Trending content ideas are scored using recency and engagement metrics collected during the scraping process. This numeric scoring helps prioritize fresh AI topics in the daily review markdown.