skills.rest
ruiyangruiyi
ruiyangruiyiCOMMUNITY
Explore all skills by this author →
GitHub

optimizing-attention-flash

Optimize transformer attention with FlashAttention-2/3 for reduced GPU memory and faster long-sequence workloads.

Updated May 26, 2026
Search Tags:#memory efficiency#transformers#flash attention#long context#gpu optimization#pytorch sdpa#h100 fp8
One-click install
npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill optimizing-attention-flash-ruiyangruiyi
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: optimizing-attention-flash
Source: https://github.com/ruiyangruiyi/hermes-agent/tree/main/optional-skills/mlops/flash-attention
Command: npx skills add https://github.com/ruiyangruiyi/hermes-agent --skill optimizing-attention-flash-ruiyangruiyi

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires flash-attn, torch, transformers, and includes references (resource) components.

What problem does it solve?

Flash Attention reduces transformer attention memory blow-ups and improves throughput so you can train or run long-context models without hitting GPU OOM or slow attention kernels.

Core Features & Use Cases

  • Faster, memory-efficient attention: Implements IO-aware tiling/recomputation to avoid materializing the full attention matrix, delivering multi-x speedups on long sequences.
  • Drop-in PyTorch SDPA support: Uses PyTorch scaled_dot_product_attention (native backend) for simpler integration (PyTorch 2.2+).
  • Advanced kernels via flash-attn: Enables FlashAttention-2 and FlashAttention-3 features such as multi-query attention, sliding window attention, and H100 FP8 acceleration for further gains.

Use case example: You’re running a Llama/Mistral fine-tuning or inference job with 8K–32K tokens and standard attention either OOMs or becomes too slow; applying Flash Attention lets the workload fit and run significantly faster while preserving results closely (typical small numerical differences).

Quick Start

Install flash-attn and switch your model’s attention to FlashAttention-2 (or native SDPA) to benchmark speed and memory on your target sequence length.

Frequently Asked Questions about optimizing-attention-flash

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I fix GPU out of memory errors when running long context transformers?▼

Flash Attention fixes GPU out of memory errors by using IO-aware tiling to avoid materializing the full attention matrix, drastically reducing memory overhead for long context transformer workloads.

Does PyTorch SDPA support FlashAttention-2 for faster inference?▼

Yes, PyTorch SDPA supports FlashAttention-2 through the native scaled_dot_product_attention backend, allowing you to switch your model's attention implementation for faster inference and simplified integration.

Can I use H100 FP8 acceleration with flash-attn for transformer training?▼

Yes, you can use H100 FP8 acceleration with flash-attn by enabling FlashAttention-3 features, which provides further speed gains during transformer training on compatible NVIDIA H100 hardware.

What is the best way to speed up Llama or Mistral fine-tuning on 8K tokens?▼

The best way to speed up Llama or Mistral fine-tuning on 8K tokens is applying Flash Attention, which lets the workload fit in GPU memory and run significantly faster while preserving results closely.

How to switch my model's attention to FlashAttention-2 in PyTorch?▼

To switch your model's attention to FlashAttention-2 in PyTorch, install the flash-attn library and update your model configuration's attn_implementation parameter, then benchmark speed and memory on your target sequence length.

Are there numerical differences when using Flash Attention for long sequences?▼

Yes, there are typical small numerical differences when using Flash Attention for long sequences, but it preserves results closely while delivering multi-x speedups and avoiding GPU OOM errors.

Related Skills

Explore More Software Engineering Skills→
plurigridplurigrid
60

system2-attention

Apply System2-attention to regenerate context and refine transformer attention weights.

Official
Advanced
qcmuuqcmuu

optimizing-attention-flash

Replace standard transformer attention with Flash Attention kernels to reduce GPU memory usage.

Community
Intermediate
Supporter09Supporter09

optimizing-attention-flash

Accelerate transformer attention and reduce GPU memory usage in PyTorch and Hugging Face Transformers.

Community
Advanced
JamesFincherJamesFincher

optimizing-attention-flash

Optimize transformer attention with Flash Attention to reduce GPU memory usage.

Community
Advanced