llm-relevance-scoring

Define an explicit five-level rubric with observable criteria, build 5-10 few-shot calibration examples spanning the score range, then prompt the model at temperature 0.1-0.3 to return a JSON score and reasoning. Calibrate against 20 human-scored items before batch processing.

Authority peaks require both a high relevance score (typically 75+) and knowledge-asymmetry signals such as original analysis, first-hand expertise, or corrections of misconceptions. Length and upvotes are unreliable proxies and should not be used.

Local Ollama models like Llama 3.1 8B are fast, private, and free for bulk scoring but may miss nuanced authority signals. GPT-4 or Claude APIs offer better calibration and human agreement at the cost of rate limits and data leaving your machine.

Score variance usually comes from high temperature settings or missing few-shot examples. Use temperature 0.1-0.3, include calibration examples in every prompt, and require chain-of-thought reasoning before the numeric score.

Avoid LLM scoring for corpora under 20 items where manual review is faster, for non-textual content without transcription, for sub-second real-time latency needs, or when a simple keyword or regex filter would suffice.

Manually score 20 items and compute Cohen's kappa on binned scores, targeting above 0.7 with mean absolute error under 15. Cross-validate a 10% sample with a second model expecting Pearson r above 0.75, and human-review 5-10% of results.