Benchmark
Measured Run 20261105-0038-d71e · dataset k8s-eval-v0: 55 questions, 55 human-reviewed · snapshot k8s-20261030-878ad3 · generation gpt-6-luna · judge gpt-6-luna. Differences show paired bootstrap 95% confidence intervals.
| Metric | n | Baseline | Intermediate | Advanced | Intermediate − Baseline | Advanced − Intermediate |
|---|---|---|---|---|---|---|
| Hit Rate@5 | 46 | 0.804 | 0.848 | 0.935 | +0.043 (-0.065, 0.174) | +0.087 (0.022, 0.174) |
| Recall@5 | 46 | 0.750 | 0.801 | 0.902 | +0.051 (-0.065, 0.174) | +0.101 (0.029, 0.188) |
| Recall@5 (multi-section) | 9 | 0.778 | 0.648 | 0.833 | -0.130 (-0.317, 0.000) | +0.185 (0.000, 0.436) |
| Full Coverage@5 (multi-section) | 9 | 0.556 | 0.444 | 0.667 | -0.111 (-0.375, 0.000) | +0.222 (0.000, 0.556) |
| MRR@10 | 46 | 0.636 | 0.651 | 0.870 | +0.015 (-0.093, 0.122) | +0.219 (0.134, 0.312) |
| nDCG@10 | 46 | 0.572 | 0.596 | 0.735 | +0.024 (-0.047, 0.098) | +0.139 (0.082, 0.201) |
| Answer Correctness | 46 | 0.804 | 0.804 | 0.913 | +0.000 (-0.136, 0.136) | +0.109 (0.022, 0.208) |
| Faithfulness | 44 | 0.992 | 0.992 | 1.000 | -0.000 (-0.023, 0.022) | +0.008 (0.000, 0.025) |
| Citation Correctness (provisional) | 25 | 0.680 | 0.643 | 0.821 | -0.037 (-0.273, 0.195) | +0.179 (-0.101, 0.444) |
| Uncited Claim Rate | 68 | 0.632 | 0.600 | 0.611 | -0.032 (-0.149, 0.082) | +0.011 (-0.117, 0.140) |
| Correct Abstention Rate | 9 | 0.889 | 1.000 | 1.000 | +0.111 (0.000, 0.364) | +0.000 (0.000, 0.000) |
| False Abstention Rate | 46 | 0.065 | 0.065 | 0.022 | +0.000 (-0.087, 0.085) | -0.043 (-0.132, 0.043) |
| Faithful but incorrect | 43 | 0.023 | 0.000 | 0.000 | -0.023 (-0.075, 0.000) | +0.000 (0.000, 0.000) |
| Context tokens (mean) | 55 | 1,684 | 1,124 | 1,230 | -560.218 (-621.183, -500.273) | +106.000 (46.343, 165.545) |
| Latency p50 (ms) | 55 | 1,928 | 1,740 | 3,385 | ||
| Latency p95 (ms) | 55 | 4,648 | 3,803 | 68,473 |
Answer metrics are LLM-judged and validated against human labels on a sample: correctness 94% agreement (n=18), faithfulness 100% agreement (n=28), citation 32% agreement (n=28).