custom-dataset-seeds

What problem does it solve?

It converts local files and user-provided datasets (like PDFs, CSVs, and text) into Lightning Rod “seeds” so you can quickly build labeled forecasting datasets without manual preprocessing.

Core Features & Use Cases

• File-to-samples ingestion: Chunk documents or map CSV columns into model-ready samples with optional embedded labels and metadata.
• FileSet-based workflows: Upload large or metadata-rich corpora as a FileSet for scalable transformation and temporal/metadata filtering.
• Flexible context and labeling strategies: Generate seeds-only, whole-document (non-RAG) context/labels using chronological constraints, or RAG context/labels using vector retrieval with payload/temporal filters.
• Fitness + chunking guidance: Provide practical checks (volume, date coverage, text quality, label availability) and recommended chunking parameters to improve results.

Quick Start

Convert your PDFs into seeds and run a transforms pipeline with a limit on generated questions by asking: “Ingest data/*.pdf as samples with chunk_size=1000 and chunk_overlap=100, create an input_dataset from those samples, then run lr.transforms.run(pipeline, input_dataset=input_dataset, max_questions=10).”