skills.rest
eddielueng
eddieluengCOMMUNITY
Explore all skills by this author →
GitHub

optimizing-attention-flash

Optimize transformer attention with PyTorch SDPA or flash-attn for long sequences.

1|Updated Apr 14, 2026
Search Tags:#memory efficiency#transformers#fp8#flash attention#long context#gpu optimization#pytorch sdpa
One-click install
npx skills add https://github.com/eddielueng/hermes-agent-zh --skill optimizing-attention-flash-eddielueng
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: optimizing-attention-flash
Source: https://github.com/eddielueng/hermes-agent-zh/tree/main/optional-skills/mlops/flash-attention
Command: npx skills add https://github.com/eddielueng/hermes-agent-zh --skill optimizing-attention-flash-eddielueng

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires flash-attn, torch, transformers, and includes references (resource) components.

What problem does it solve?

Flash Attention reduces transformer attention latency and GPU memory usage so long-context models can run faster and avoid out-of-memory errors.

Core Features & Use Cases

  • 2–4x speedup and 10–20x memory reduction for attention by using IO-aware tiling and recomputation rather than materializing the attention matrix.
  • Drop-in use with PyTorch SDPA (PyTorch 2.2+) via scaled_dot_product_attention to automatically leverage Flash Attention when available.
  • Advanced support with flash-attn including multi-query attention, sliding window attention, and FlashAttention-3 optimizations (notably FP8 on H100).
  • Use case: Running a Llama/Mistral-style model with long sequences (e.g., 2K–16K tokens) that OOMs with standard attention, while you need faster inference or training throughput.

Quick Start

Enable Flash Attention in your PyTorch 2.2+ model by using torch.nn.functional.scaled_dot_product_attention(q, k, v) and letting the backend select the Flash kernel when available.

Frequently Asked Questions about optimizing-attention-flash

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix out-of-memory errors when running long context transformers?▼

Flash Attention fixes out-of-memory errors in long context transformers by using IO-aware tiling and recomputation, achieving a 10–20x memory reduction without materializing the full attention matrix.

Can I use PyTorch scaled_dot_product_attention to speed up model inference?▼

Yes, PyTorch scaled_dot_product_attention in PyTorch 2.2+ enables Flash Attention automatically when available, providing a drop-in method to speed up inference and reduce latency for long sequences.

Does flash-attn support FP8 on H100 GPUs for faster attention?▼

FlashAttention-3 supports FP8 on H100 GPUs through the flash-attn library, offering advanced optimizations including multi-query attention and sliding window attention for further throughput gains.

What is the best way to optimize attention for sequences above 512 tokens?▼

Optimizing attention for sequences above 512 tokens is best achieved using Flash Attention, which applies IO-aware tiling to deliver 2–4x speedups and prevent latency bottlenecks during training and inference.

How does Flash Attention reduce attention latency for Llama models?▼

Flash Attention reduces latency for Llama models by avoiding the materialization of the attention matrix, instead using tiling and recomputation to process 2K–16K token sequences with significantly lower GPU memory usage.

Do I need to install flash-attn to use sliding window attention in PyTorch?▼

Yes, installing the flash-attn library is required to access advanced features like sliding window attention and FlashAttention-3 optimizations, whereas standard PyTorch SDPA handles basic Flash Attention backend dispatching.

Related Skills

Explore More Software Engineering Skills→
ldzhhxxldzhhxx
1

optimizing-attention-flash

Optimize transformer attention with Flash Attention for reduced memory usage.

Community
Advanced
SignmanalSignmanal
1

optimizing-attention-flash

Optimize scaled dot-product attention for long transformer sequences on GPUs.

Community
Intermediate
Supporter09Supporter09

optimizing-attention-flash

Accelerate transformer attention and reduce GPU memory usage in PyTorch and Hugging Face Transformers.

Community
Advanced
JamesFincherJamesFincher

optimizing-attention-flash

Optimize transformer attention with Flash Attention to reduce GPU memory usage.

Community
Advanced