attention-residuals

What problem does it solve?

Transformer models often suffer from unstable training and poor convergence due to standard residual connections. Attention Residuals (AttnRes) reframe how residuals are integrated into attention blocks to improve gradient flow and training robustness.

Core Features & Use Cases

• Redesigns the residual path in attention layers to enhance stability and convergence.
• Applicable to large language models, vision transformers, and multimodal architectures to boost training robustness.
• Use case: upgrade an existing Transformer-based model to achieve more stable early training and better final performance on long sequences.

Quick Start

Enable AttnRes in a Transformer block to improve stability during training.