skills.rest
alirezarezvani
alirezarezvaniCOMMUNITY
Explore all skills by this author →
Website𝕏@RezaRezvaniBlnGitHub

spinning-up-deep-rl

Applies Joshua Achiam's Spinning Up frameworks to deep RL algorithm selection, debugging, and experimentation.

25.3k|3.6k|Updated Oct 19, 2025
Search Tags:#Debugging#ppo#SAC#deep rl#Reinforcement Learning#policy gradient#spinning up
One-click install
npx skills add https://github.com/alirezarezvani/claude-skills --skill spinning-up-deep-rl
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: spinning-up-deep-rl
Source: https://github.com/alirezarezvani/claude-skills/tree/main/engineering/spinning-up-deep-rl/skills/spinning-up-deep-rl
Command: npx skills add https://github.com/alirezarezvani/claude-skills --skill spinning-up-deep-rl

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Deep RL practitioners struggle with choosing between algorithms like PPO, SAC, and TD3, debugging code that fails silently, and running experiments rigorous enough to trust. This Skill packages Joshua Achiam's Spinning Up in Deep RL documentation into a queryable knowledge base of frameworks, chapter summaries, and decision rules.

Core Features & Use Cases

  • Algorithm Selection Guidance: Place any RL algorithm using the model-free taxonomy, the on-policy vs off-policy trade-off, and decision tables covering VPG, TRPO, PPO, DDPG, TD3, and SAC.
  • Silent-Failure Debugging: Diagnose RL code that runs but never learns using shape-mismatch checks, instrumentation checklists, and the policy-gradient-loss-is-not-a-loss warning.
  • Rigorous Experiment Design: Apply the four rigor standards — tuned baselines, 3-10+ seeds, precommitted final runs, and per-claim ablations — plus cheatsheet thresholds and Spinning Up CLI commands.
  • Use Case: A researcher whose PPO training curve is flat asks why; the Skill routes to the debugging frameworks, flags the silent-failure archetypes, and recommends instrumentation and toy-environment iteration under 5 minutes.

Quick Start

Ask the agent to explain the difference between TRPO and PPO and which one to use for a continuous control task with limited environment samples.

Frequently Asked Questions about spinning-up-deep-rl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose between PPO, SAC, TD3, and DDPG for a reinforcement learning task?▼

Choose by action space and priority: PPO for discrete actions or maximum stability, SAC or TD3 for continuous control when sample efficiency matters most. DDPG, TD3, and the shipped SAC are continuous-only, and SAC adds entropy regularization for explicit explore-exploit control.

Why is my reinforcement learning agent not learning even though the code runs?▼

Broken RL code almost always fails silently, so assume a bug before tuning hyperparameters. Check tensor shapes first — a [N] vs [N,1] mismatch broadcasts without errors and corrupts the Bellman backup — then verify the loss uses the right equation, distribution, and data.

What is the difference between on-policy and off-policy reinforcement learning?▼

On-policy algorithms like VPG, TRPO, and PPO update only from data collected by the current policy, giving stability but poor sample efficiency. Off-policy methods like DDPG, TD3, and SAC reuse all past data via a replay buffer, gaining sample efficiency but with more failure modes.

How many random seeds do I need for a rigorous RL experiment?▼

Run at least 3 seeds, and 10 or more to be thorough, because deep RL is highly seed-sensitive and two seed groups can look like different distributions. Plot the mean with a standard deviation band, and precommit to reporting fresh final runs rather than your best runs.

Can I use the policy gradient loss curve to monitor training progress?▼

No — the policy gradient loss is not a performance measure and can go to negative infinity while performance collapses. Only average return means anything, so monitor AverageEpRet and instrument value estimates, episode lengths, and policy entropy instead.

What topics does Spinning Up in Deep RL not cover?▼

It excludes DQN and discrete-action value learning, recurrent or convolutional architectures, partial observability, model-based implementations, and deep RL work after early 2020. The six implementations are educational MLP actor-critics for fully-observed, non-image environments.

Related Skills

Explore More Education & Research Skills→
Ritabrata-ChakrabortyRitabrata-Chakraborty

stable-baselines3

Train single-agent reinforcement learning models in Gymnasium environments with PyTorch.

Community
Advanced
Embers-of-the-FireEmbers-of-the-Fire

experiment-design

Plan multi-stage research experiments with baselines, datasets, hyperparameter grids, and metrics.

Community
Advanced
wildwasserwildwasser
35

ml-experiment

Track machine learning experiments with structured logging and reproducibility.

Community
Advanced
hxk622hxk622
8

stable-baselines3

Train reinforcement learning agents using stable-baselines3 algorithms.

Community
Intermediate