index-lifecycle

Document and modify Searchlite's WAL, segment, and manifest lifecycle.

Updated Dec 20, 2025

Search Tags:#indexing #manifest #compaction #segment #durability #WAL #Searchlite

Ready to run

npx skills add https://github.com/davidkelley/searchlite --skill index-lifecycle

💬 Prompt for ChatGPT / Claude Chat▼

Please help me install this Agent Skill.
Skill: index-lifecycle
Source: https://github.com/davidkelley/searchlite/tree/main/.codex/skills/index-lifecycle
Command: npx skills add https://github.com/davidkelley/searchlite --skill index-lifecycle

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

This skill helps developers understand and modify the storage/indexing pipeline of Searchlite, including the WAL, segment builds, and manifest-based durability and compaction, guiding changes to ingest, fast fields, and segment layouts.

Core Features & Use Cases

Clear mental model of how a single writer, WAL, segments, and manifest interact during Init, Ingest, Commit, and Refresh.
Guidance for maintaining durability guarantees, atomic commits, and segment compaction strategies.
Use Case: when implementing a new ingest flow or updating backends, reference this lifecycle cheat sheet to ensure correctness and performance.

Quick Start

Review the lifecycle steps and apply them to a sample ingest workflow: initialize an index, ingest docs, commit to create a new segment, and observe the manifest update.

Frequently Asked Questions about index-lifecycle

High-intent search queries and answers about installing and using this skill.

FAQPage Schema

How does the WAL to segment to manifest lifecycle work in Searchlite?▼

The Searchlite lifecycle moves data through Init, Ingest, Commit, Refresh, and Maintain phases. A single writer uses the WAL for durability, builds segments during commits, and updates the manifest to track segment layout. Compaction merges segments to optimize storage and query performance.

How do I ensure atomic commits when modifying the index lifecycle in Searchlite?▼

To ensure atomic commits in Searchlite, the lifecycle skill guides you through the WAL write, segment build, and manifest update sequence. This guarantees durability by making the manifest the single source of truth, ensuring incomplete segments are never referenced.

What is the best way to implement a new ingest flow in Searchlite?▼

When implementing a new ingest flow in Searchlite, reference the lifecycle steps from Init through Ingest and Commit. This ensures correct WAL usage, proper fast fields handling, and accurate segment creation before the manifest update finalizes the operation.

When do I need to trigger segment compaction in Searchlite?▼

Segment compaction in Searchlite is handled during the Maintain phase. You trigger it to merge smaller segments created during Ingest and Commit, reducing overhead and ensuring fast fields remain efficient for queries across the index manifest.

Does Searchlite support updating fast fields during the index refresh phase?▼

Searchlite updates fast fields during segment builds as part of the Ingest and Commit phases. The Refresh phase makes these newly committed segments visible to readers, ensuring fast fields are available for queries without interrupting the ingest pipeline.