long-context

Extend transformer context windows using RoPE, YaRN, ALiBi, and position interpolation techniques.

13.0k|930|Updated Nov 3, 2025

Search Tags:#yarn #position interpolation #rope #long context #llm fine-tuning #context extension #alibi

Ready to run

npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill long-context-orchestra-research

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: long-context
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/19-emerging-techniques/long-context
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill long-context-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, flash-attn, and includes references (resource) components.

What problem does it solve? Pre-trained transformer models like LLaMA and Mistral are limited to their original context windows (e.g., 2048 tokens), making them unable to process long documents such as books, codebases, or extended conversations without expensive retraining. ## Core Features & Use Cases - Positional Encoding Implementations: Provides working code for RoPE (Rotary Position Embeddings) and ALiBi (Attention with Linear Biases) for new model architectures. - Context Extension Methods: Covers YaRN, Position Interpolation, and linear/dynamic RoPE scaling to extend existing models to 32k-128k+ tokens with minimal fine-tuning (400-1000 steps). - Fine-tuning & Evaluation Guides: Includes data preparation, training configurations, perplexity evaluation, and passkey retrieval testing for long-context models. - Use Case: Extend a LLaMA-2-7B model from 2k to 32k tokens by configuring YaRN rope_scaling and fine-tuning for 400 steps on long documents from PG-19. ## Quick Start Extend my LLaMA-2-7B model to handle 32k token documents using YaRN scaling and show me the fine-tuning configuration.

Frequently Asked Questions about long-context

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How do I extend LLaMA context length to 32k tokens?▼

Set rope_scaling in the model config with type "linear" or "yarn" and a factor of 16.0, then fine-tune for 400-1000 steps on long documents. YaRN achieves better quality with only 400 steps and 10x less training data than position interpolation.

What is the difference between YaRN and position interpolation?▼

YaRN uses NTK-aware interpolation that treats high and low frequencies differently plus attention temperature scaling, while position interpolation uniformly compresses all positions. YaRN needs 2.5x fewer steps and extrapolates better to longer contexts.

RoPE vs ALiBi: which positional encoding should I use?▼

ALiBi is best for training new models from scratch since it offers unlimited context, 11% less memory, and strong extrapolation. RoPE with YaRN scaling is better for extending existing pre-trained models like LLaMA that already use rotary embeddings.

Does position interpolation work without fine-tuning?▼

No, applying rope_scaling without fine-tuning produces poor results. Position interpolation requires roughly 1000 fine-tuning steps on long documents matching the target context length to restore model quality.

Why does long-context training run out of memory?▼

Attention memory grows quadratically with sequence length. Enable gradient checkpointing, use Flash Attention 2 via attn_implementation="flash_attention_2", reduce batch size to 1, and use bfloat16 precision to fit 32k sequences in memory.

What are the limitations of context extension methods?▼

Aggressive scaling factors like 16x require more fine-tuning data and steps, and position interpolation extrapolates poorly beyond the trained range. Incremental scaling (8k to 16k to 32k) with fine-tuning at each stage gives more stable results.