skills.rest
loteiron
loteironCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

flash-attention

Implement memory-efficient Flash Attention to accelerate transformer training and inference on GPUs.

Updated Sep 10, 2026
Search Tags:#PyTorch#Transformers#Flash Attention#memory efficiency#FP8#long context#gpu optimization
One-click install
npx skills add https://github.com/loteiron/ZeusAgent --skill flash-attention-loteiron
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flash-attention
Source: https://github.com/loteiron/ZeusAgent/tree/main/optional-skills/mlops/flash-attention
Command: npx skills add https://github.com/loteiron/ZeusAgent --skill flash-attention-loteiron

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires flash-attn, torch, transformers, and includes references (resource) components.

What problem does it solve? Standard transformer attention materializes an O(N²) attention matrix, causing out-of-memory errors and slow throughput on long sequences. This Skill guides you through enabling Flash Attention in PyTorch or the flash-attn library to get 2-4x speedups and up to 94% memory reduction without accuracy loss. ## Core Features & Use Cases - PyTorch native integration: Enable Flash Attention via F.scaled_dot_product_attention and the sdpa_kernel backend on PyTorch 2.2+. - flash-attn library workflows: Use advanced features like multi-query attention, sliding window attention, and causal masking. - H100 FP8 optimization: Build FlashAttention-3 from the hopper/ source directory for FP8 forward passes on Hopper GPUs. - Use Case: Fine-tuning a Llama 2 7B model with 8K context that OOMs with standard attention — enable Flash Attention 2 in HuggingFace Transformers with attn_implementation="flash_attention_2" to fit training on an A100. ## Quick Start Enable Flash Attention in my PyTorch transformer model and verify the speedup with a benchmark against standard attention.

Frequently Asked Questions about flash-attention

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I enable Flash Attention in PyTorch?▼

Use torch.nn.functional.scaled_dot_product_attention, which automatically selects Flash Attention when available on PyTorch 2.2+. To force the backend, wrap the call in torch.nn.attention.sdpa_kernel with SDPBackend.FLASH_ATTENTION.

How to use Flash Attention with HuggingFace Transformers?▼

Pass attn_implementation="flash_attention_2" to AutoModel.from_pretrained along with torch_dtype=torch.float16. Transformers 4.36+ supports this natively for Llama, Mistral, Falcon, GPT-NeoX, Phi, Qwen, and other architectures.

Flash Attention vs xFormers: which should I use?▼

Use Flash Attention when you need 2-4x speedup and large memory reduction for standard attention patterns on GPU. Choose xFormers when you need more attention variants beyond speed optimization, and use memory-efficient attention for CPU inference.

Does Flash Attention work on V100 or older GPUs?▼

No, Flash Attention requires NVIDIA Turing (compute capability 7.5) or newer, with full support on Ampere GPUs like A100 and A10. V100 (Volta) and CPU inference are not supported.

Why is Flash Attention not giving me a speedup?▼

Speedup scales with sequence length: under 512 tokens you may see only 10-20% improvement, while 2K+ tokens yield 3-4x. Also verify inputs are float16 or bfloat16, since float32 is not supported.

Does pip install flash-attn include FlashAttention-3 FP8 support?▼

No, the pip flash-attn package ships FlashAttention-2 only. FlashAttention-3 with FP8 forward is a separate beta built from source in the repo's hopper/ directory and exposed via the flash_attn_interface module.

Related Skills

Explore More Software Engineering Skills→
NousResearchNousResearch
239.1k

flash-attention

Accelerate transformer attention with IO-aware tiling for faster training and inference.

Official
Intermediate
Prathmesh2000Prathmesh2000
1

ml-engineer

Trains and deploys production ML systems with scalable pipelines and MLOps practices.

Community
Advanced
Mathews-TomMathews-Tom
310

gpu-optimizer

Optimize PyTorch and CUDA training on NVIDIA GPUs with 8–24GB VRAM.

Community
Advanced
AlvaroBianoAlvaroBiano

optimizing-attention-flash

Optimizes transformer attention for memory-efficient training and inference on long sequences.

Community
Advanced