skills.rest
MLT-OSS
MLT-OSSOFFICIAL
Explore all skills by this author →
WebsiteGitHub

optimizing-attention-flash

Optimizes transformer attention with Flash Attention for faster training and lower GPU memory usage.

14|5|Updated Apr 9, 2026
Search Tags:#pytorch#memory efficiency#transformers#fp8#flash attention#long context#gpu optimization
One-click install
npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill optimizing-attention-flash-mlt-oss
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: optimizing-attention-flash
Source: https://github.com/MLT-OSS/hermes-agent-go/tree/main/optional-skills/mlops/flash-attention
Command: npx skills add https://github.com/MLT-OSS/hermes-agent-go --skill optimizing-attention-flash-mlt-oss

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires flash-attn, torch, transformers, and includes references (resource) components.

What problem does it solve? Standard transformer attention consumes O(N²) memory and becomes prohibitively slow or runs out of GPU memory when training or running inference on sequences longer than a few thousand tokens. ## Core Features & Use Cases - PyTorch Native Integration: Enable Flash Attention through torch.nn.functional.scaled_dot_product_attention on PyTorch 2.2+ with no extra dependencies. - flash-attn Library Workflows: Use advanced features like multi-query attention, sliding window attention, and causal masking via the flash-attn package. - H100 FP8 Optimization: Run FlashAttention-3 with FP8 precision on H100 GPUs for up to 1.2 PFLOPS throughput. - Use Case: When fine-tuning Llama 2 7B with 8K context hits CUDA OOM on an A100, switch the attention implementation to Flash Attention to cut memory by over 50% and gain a 3-4x training speedup. ## Quick Start Replace my model's standard attention with Flash Attention using PyTorch SDPA and verify the speedup with a benchmark.

Frequently Asked Questions about optimizing-attention-flash

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I enable Flash Attention in PyTorch?▼

Use torch.nn.functional.scaled_dot_product_attention with PyTorch 2.2 or later, which automatically dispatches to Flash Attention kernels when available. You can force the Flash backend using torch.backends.cuda.sdp_kernel with enable_flash=True.

How to use Flash Attention with HuggingFace Transformers models?▼

Pass attn_implementation="flash_attention_2" to AutoModel.from_pretrained along with torch_dtype=torch.float16. Transformers 4.36+ supports this for Llama, Mistral, Falcon, Qwen, Gemma, and many other architectures.

Flash Attention vs xFormers: which should I use?▼

Flash Attention is the better choice when you need raw speed and memory efficiency for standard attention patterns on GPU. xFormers offers a wider variety of attention variants, so use it when you need specialized attention mechanisms beyond fast exact attention.

Does Flash Attention work on V100 GPUs or CPU?▼

No. Flash Attention requires NVIDIA Turing (T4) or newer, with full support on Ampere GPUs like A100 and A10. V100 (Volta) and CPU inference are not supported; use standard or memory-efficient attention instead.

Why is Flash Attention not giving me a speedup?▼

Speedup scales with sequence length: sequences under 512 tokens see only 10-20% improvement, while 2K+ tokens see 3-4x. Also verify your tensors are float16 or bfloat16, since float32 is not supported by Flash Attention kernels.

Why does flash-attn installation fail with ImportError?▼

Install with pip install flash-attn --no-build-isolation to avoid build environment issues. If it still fails, install the CUDA toolkit first (e.g., conda install cuda -c nvidia) and ensure CUDA 11.8 or later is available.

Related Skills

Explore More Software Engineering Skills→
AlvaroBianoAlvaroBiano

optimizing-attention-flash

Optimizes transformer attention for memory-efficient training and inference on long sequences.

Community
Advanced
stephanjstephanj
9

fine-tuning-expert

Fine-tune Large Language Models with LoRA and QLoRA techniques.

Community
Advanced
Mathews-TomMathews-Tom
310

gpu-optimizer

Optimize PyTorch and CUDA training on NVIDIA GPUs with 8–24GB VRAM.

Community
Advanced
nelohenriqnelohenriq
1

optimizing-attention-flash

Integrate Flash Attention variants into PyTorch transformer models for NVIDIA GPUs.

Community
Advanced