skills.rest
vivekgoquest
vivekgoquestCOMMUNITY
Explore all skills by this author →
GitHub

flash-attention

Implement Flash Attention in PyTorch transformers for faster long-sequence training and inference.

Updated Aug 22, 2026
Search Tags:#PyTorch#Transformer#Flash Attention#memory efficiency#long context#gpu optimization#training speedup
One-click install
npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill flash-attention-vivekgoquest
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flash-attention
Source: https://github.com/vivekgoquest/hermes-agent-stable/tree/main/optional-skills/mlops/flash-attention
Command: npx skills add https://github.com/vivekgoquest/hermes-agent-stable --skill flash-attention-vivekgoquest

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires flash-attn, torch, transformers, and includes references (resource) components.

What problem does it solve? Standard transformer attention consumes O(N²) memory and becomes prohibitively slow or runs out of GPU memory on long sequences, blocking training and inference beyond a few thousand tokens. ## Core Features & Use Cases - PyTorch SDPA Integration: Enable Flash Attention via torch.nn.functional.scaled_dot_product_attention with backend selection and verification steps. - flash-attn Library Workflows: Use flash_attn_func for causal masking, multi-query attention, sliding window attention, and dropout control. - H100 FP8 Optimization: Build FlashAttention-3 from source for FP8 forward passes on Hopper GPUs. - Use Case: Fine-tuning a Llama 2 7B model with 8K context that OOMs with standard attention — enable flash_attention_2 in HuggingFace Transformers to fit training on an A100 with a 2-3x throughput gain. ## Quick Start Enable Flash Attention in my PyTorch transformer model and verify the speedup and accuracy against the standard attention baseline.

Frequently Asked Questions about flash-attention

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I enable Flash Attention in PyTorch?▼

Use torch.nn.functional.scaled_dot_product_attention with PyTorch 2.2+, which automatically dispatches to Flash Attention when available. To force the backend, wrap the call in torch.nn.attention.sdpa_kernel with SDPBackend.FLASH_ATTENTION.

How to use Flash Attention with HuggingFace Transformers?▼

Pass attn_implementation="flash_attention_2" to AutoModel.from_pretrained along with torch_dtype=torch.float16. Transformers 4.36+ supports this natively for Llama, Mistral, Falcon, Qwen, Gemma, and other architectures.

Flash Attention vs xFormers: which should I use?▼

Use Flash Attention when you need 2-4x speedup and large memory reduction for standard attention on GPU. Choose xFormers when you need more attention variants beyond speed, and use memory-efficient attention for CPU inference since Flash Attention requires a GPU.

Does Flash Attention work on V100 or older GPUs?▼

No, Flash Attention requires NVIDIA Turing (compute capability 7.5) or newer, with full support on Ampere GPUs like A100 and A10. V100 (Volta) and CPU inference are not supported.

Why is Flash Attention not speeding up my model?▼

Speedup scales with sequence length: under 512 tokens gains are only 10-20%, while 2K+ tokens see 3-4x. Also verify inputs are float16 or bfloat16, since float32 is not supported by Flash Attention kernels.

Does pip install flash-attn include FlashAttention-3 FP8 support?▼

No, the pip package ships FlashAttention-2 only. FlashAttention-3 with FP8 forward is a separate beta built from source in the repo's hopper/ directory and exposed via the flash_attn_interface module, requiring an H100 GPU.

Related Skills

Explore More Software Engineering Skills→
loteironloteiron

flash-attention

Implement memory-efficient Flash Attention to accelerate transformer training and inference on GPUs.

Community
Intermediate
guccangguccang
3

serving-llms-vllm

Deploy vLLM inference endpoints with OpenAI-compatible APIs and GPU optimization.

Community
Advanced
Mathews-TomMathews-Tom
310

gpu-optimizer

Optimize PyTorch and CUDA training on NVIDIA GPUs with 8–24GB VRAM.

Community
Advanced
AlvaroBianoAlvaroBiano

optimizing-attention-flash

Optimizes transformer attention for memory-efficient training and inference on long sequences.

Community
Advanced