What problem does it solve?
This Skill turns scored, discrete-action decision problems into an efficient training loop by learning a value function while using a progressive, search-guided policy improvement strategy.
Core Features & Use Cases
- Dense score-based learning: builds a Q-style value head from a per-state
evaluate(state) correlate instead of relying only on sparse terminal rewards.
- Russian Doll MCTS with value-head leaves: runs progressive narrowing search so wide action spaces remain tractable, using the network (and a heuristic fallback) to evaluate search leaves.
- AHA mistake correction: detects evaluation drops after a chosen action during training and applies immediate corrective replay signal.
- Training progress annealing: anneals MCTS iteration counts, exploration, and funnel widths as the value function becomes more reliable.
Use it for environments like board games, turn-based strategy, or any simulation where you can enumerate discrete actions, encode state tensors, and compute a current-player-perspective scalar score that correlates with ultimate success.
Quick Start
Use the deep-q-rl skill to train an agent by implementing the ScoredEnvironment interface with encode_state, evaluate, legal_actions, apply, and is_terminal for your environment, then run self-play or rollout-based training with Russian Doll MCTS and AHA enabled for training.