Verified:GitHub

optimizing-attention-flash

Optimize transformer attention to reduce GPU memory usage and accelerate computations.

Updated Apr 12, 2026

Search Tags:#pytorch #fp8 #long-context #flash-attention #h100 #flash-attn #attention-optimization

One-click install

npx skills add https://github.com/DaddyElonMusk69/motis-agent --skill optimizing-attention-flash-daddyelonmusk69

Or copy as Structured Prompt for Agent▼

Please help me install this Agent Skill.
Skill: optimizing-attention-flash
Source: https://github.com/DaddyElonMusk69/motis-agent/tree/main/optional-skills/mlops/flash-attention
Command: npx skills add https://github.com/DaddyElonMusk69/motis-agent --skill optimizing-attention-flash-daddyelonmusk69

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Large-sequence transformer attention consumes O(N²) memory and can cause GPU out-of-memory errors and slow training/inference for contexts beyond a few hundred tokens. This Skill guides engineers to replace standard attention with Flash Attention or PyTorch's native SDPA to drastically reduce memory footprint and accelerate attention computation.

Core Features & Use Cases

Memory-efficient attention: Eliminates materializing the full attention matrix to cut memory usage by an order of magnitude for long contexts.
Speed improvements: Typical 2-4x speedup for sequences >512 tokens and higher gains on H100 with FP8.
Compatibility & advanced modes: Instructions for PyTorch native SDPA (2.2+), the flash-attn library for sliding windows and multi-query attention, and FP8/H100 optimizations.
Practical workflows: Checklists for integrating into existing models, benchmarking guidance, and troubleshooting tips for common CUDA and dtype issues.
Use case examples: training large models with long context, serving long-context inference, and enabling sliding-window or paged KV caches.

Quick Start

Enable Flash Attention by switching your model's attention implementation to PyTorch's scaled_dot_product_attention or installing and calling the flash-attn library on CUDA tensors.

Frequently Asked Questions about optimizing-attention-flash

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I optimize transformer attention to reduce GPU memory usage for long contexts?▼

Optimize transformer attention by replacing standard attention with Flash Attention or PyTorch's native SDPA. This avoids materializing the full attention matrix, drastically cutting memory usage and accelerating computation for long contexts.

What is the best way to speed up attention computation on H100 GPUs?▼

Speed up attention computation on H100 GPUs by utilizing FP8-capable kernels. This Skill provides optimization instructions for H100 FP8 workloads to achieve maximum performance gains beyond standard Flash Attention speedups.

Does PyTorch scaled_dot_product_attention support sliding-window and multi-query attention?▼

PyTorch scaled_dot_product_attention handles memory-efficient attention, while the flash-attn library specifically supports sliding-window and multi-query attention. This Skill guides integration of both for advanced attention modes.

Do I need CUDA 12.0+ to use flash-attn for attention optimization?▼

You need CUDA 11.8+ or 12.0+ together with PyTorch 2.2+ or the flash-attn library to optimize attention. Optional FP8-capable H100 kernels are required for maximum performance gains.