deep-q-rl

Train Q-networks with Russian Doll MCTS in scored discrete-action environments.

Updated Apr 20, 2026
One-click install
npx skills add https://github.com/thistleknot/skills --skill deep-q-rl
Or copy as Structured Prompt for Agent
Please help me install this Agent Skill.
Skill: deep-q-rl
Source: https://github.com/thistleknot/skills/tree/main/deep-q-rl
Command: npx skills add https://github.com/thistleknot/skills --skill deep-q-rl

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This Skill turns scored, discrete-action decision problems into an efficient training loop by learning a value function while using a progressive, search-guided policy improvement strategy.

Core Features & Use Cases

  • Dense score-based learning: builds a Q-style value head from a per-state evaluate(state) correlate instead of relying only on sparse terminal rewards.
  • Russian Doll MCTS with value-head leaves: runs progressive narrowing search so wide action spaces remain tractable, using the network (and a heuristic fallback) to evaluate search leaves.
  • AHA mistake correction: detects evaluation drops after a chosen action during training and applies immediate corrective replay signal.
  • Training progress annealing: anneals MCTS iteration counts, exploration, and funnel widths as the value function becomes more reliable.

Use it for environments like board games, turn-based strategy, or any simulation where you can enumerate discrete actions, encode state tensors, and compute a current-player-perspective scalar score that correlates with ultimate success.

Quick Start

Use the deep-q-rl skill to train an agent by implementing the ScoredEnvironment interface with encode_state, evaluate, legal_actions, apply, and is_terminal for your environment, then run self-play or rollout-based training with Russian Doll MCTS and AHA enabled for training.

Frequently Asked Questions about deep-q-rl

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I train reinforcement learning policies in environments with dense state scoring instead of sparse terminal rewards?

Deep Q-learning with Russian Doll MCTS trains policies from dense state scores by building a Q-style value head from a per-state evaluate correlate, using a target network and replay buffer to optimize decisions without waiting for sparse terminal rewards.

How does Russian Doll MCTS keep progressive narrowing search tractable for wide discrete action spaces?

Russian Doll MCTS applies progressive narrowing search to wide discrete action spaces by using action categorization with weighted sampling and evaluating search leaves with a value-head Q-network, annealing funnel widths and exploration as the value function becomes more reliable.

Can I use self-play to train a value function for turn-based strategy games if I can encode state tensors and enumerate legal actions?

Yes, self-play training works for turn-based strategy games or board games where you can encode state tensors, enumerate discrete legal actions, compute a current-player-perspective scalar score, and simulate transitions with an apply function.

What is AHA mistake correction in deep Q-learning and when does it apply corrective replay signals?

AHA mistake correction detects evaluation drops after a chosen action during training and applies immediate corrective replay signal, improving policy decisions by penalizing moves that cause immediate state score degradation during self-play rollouts.

Do I need a bounded and calibrated evaluate score to use a Q-network with Bellman targets for policy training?

Yes, a bounded and calibrated evaluate score correlate is required to train the value-head Q-network with Bellman targets, ensuring the scalar per current-player POV accurately correlates with ultimate success for progressive narrowing search optimization.

How do I start training an agent using scored discrete-action environments with deep Q-learning?

Implement the ScoredEnvironment interface with encode_state, evaluate, legal_actions, apply, and is_terminal functions for your environment, then run self-play or rollout-based training with Russian Doll MCTS and AHA mistake correction enabled for progressive policy improvement.