logprob-prefill-analysis

Prefill sensitivity analysis evaluates model susceptibility to reward hacking by applying exploit-oriented prefills, then measuring token and logprob metrics, KL divergence, and trajectory trends across checkpoints to characterize vulnerability.

You can measure reward hacking susceptibility by running the prefill evaluation scripts to compute logprob and token metrics, KL divergence, and trajectory analyses, comparing results across different exploit types and checkpoints.

Yes, this workflow requires a local or remote vLLM serving setup to compute logprob metrics, KL divergence, and trajectory analyses across your model checkpoints via the provided evaluation scripts.

Yes, comparing KL divergence and logprob trajectory metrics across checkpoints and exploit types helps identify early warning signals indicating when a model becomes easily exploitable.

The best way is to execute the end-to-end pipeline using the provided scripts to evaluate how different prefill levels influence exploitability, comparing metrics and trajectory analyses across multiple exploit types.

Limitations include the strict dependency on a configured vLLM serving environment and checkpoint access, requiring specific scripts like eval_prefill_sensitivity.py and prefill_trajectory_analysis.py to execute the full analysis.