Five mutually exclusive buckets from the judged analysis snapshot. Each one has a different owner and a different fix, so the mix, not the headline Hit@10, is what should drive the next run.
Each stage keeps only the examples that survived the previous one. The stage with the biggest drop is where the next study is worth the most.
Ordered by bucket size. Evidence is auto-computed from this split; each direction is written as a question, a study that answers it, and the claim that study would let you defend.
Where does the redundancy come from? Not from the item codes: when the ten answers do spread across more first codes, the judge sees more distinct interests and rates redundancy lower, so different first codes really do mean different products. It comes from the shape of training. Each of the 16 rollouts writes its own thinking and then one item, so training gets its variety by varying the thought. At evaluation the thought is frozen and all ten items come from one conditional distribution, and that distribution was never trained to spread.
Change the shape of the group before changing the reward. Sample a few thoughts, hold each one fixed, and sample several items under it. Those items share the evaluator's conditioning, so they are a legitimate stand-in for the decoded list, which 16 independently sampled rollouts are not. Reward that set by how many distinct interests it covers as well as by whether the target is in it, and give each item credit equal to how much the set score drops without it. Put the greedy thought in the group, since that is the one evaluation uses.
A set-level reward computed under a fixed thought raises coverage at equal or better exact match, and it does so by changing the policy, not the decoder. The supporting measurement is that training buys its diversity from the reasoning while evaluation cannot, so the number everyone reports comes from a distribution nobody trained.
How do method comparisons change when examples are stratified by how much target signal is visible in the observed history?
Learn a predictability estimator from history statistics alone, calibrated against the LLM judge's rating, and use it twice: to publish predictability-stratified leaderboards for existing baselines, and to weight the RL objective by estimated learnability. Pair it with a graded semantic reward (category, brand, embedding distance) that scores near-misses instead of zeroing them.
If stratification compresses or reorders the gaps between published methods, the field's headline metric is sensitive to test-set composition. A released predictability annotation for standard splits would make that sensitivity measurable.
Recommendation is the rare reasoning domain with a verifiable intermediate: the interest a CoT commits to can be checked against the target's own category or brand. Does process supervision on that latent beat outcome-only RL, and is the CoT causal or decorative?
Two studies. (1) Causality: counterfactually edit the CoT's stated interest, regenerate the answer, and measure how far it moves; a decorative CoT will barely shift. (2) Supervision: train a process reward on interest identification, or treat intent as a latent variable with the CoT as its amortised inference and optimise an ELBO-style objective, then compare against exact-match-only RL at matched compute.
Evidence on whether process rewards beat outcome rewards in a domain where the intermediate step is automatically checkable, plus a causal measurement of CoT faithfulness that does not rely on human raters. Both are open questions in the reasoning literature and free of label cost here.
Is the target interest merely mentioned, actually selected, or selected but omitted from the final slate? The current boolean cannot distinguish those stages.
Require the CoT to emit a ranked intent list before the SID. Allocate item samples or beam slots across those intents, then add a set-level reward for covering the selected intents with relevant items. Count it as a grounding failure only when a selected intent receives no matching item.
A structured intent-to-slate interface separates reasoning selection from candidate coverage, so each failure gets the reward and intervention it actually needs.
Green = share of that bucket where the exact target made the beam-10. Grey = how many examples sit in the bucket. All relationships are descriptive, not causal.
Repeat targets are hit 22.0% of the time, exploration targets 0.0%. The benchmark is dominated by broadening targets (38%), the hardest large class, so a model can look better simply by facing an easier intent mix. Report Hit@10 per relation, not just the average.
The two lowest predictability levels cover 1,398 examples (23%) and hit 0.5%; levels 4 and 5 hit 18.6%. Report these slices separately. The judge label identifies weak observed signal, but it does not prove irreducible error.
Among inferable targets, a CoT that identifies the target's interest hits 13.6%; one that doesn't hits 5.6%, a 2.4× gap that is far larger than any decoding knob. And with CoT↔answer consistency at 4.2/5, the decoder already follows whatever the CoT decides, so fixing the CoT should convert directly into hits.
Among inferable targets the best-performing slate covers 6–10 interests (13.2% Hit@10 over 1,769 examples), and less redundancy goes with more hits. That makes diversity a tunable reward term rather than a fixed rule: pick the per-domain target coverage, then reward the beam for hitting it.
The beam budget is 10 slots. 0.9 of them come back with an item already in the history, while only 2.9% of targets are literal repeats. Repeats are worth keeping: when the target really is a repeat the model hits it 24% of the time, just not 0.9 slots' worth. That gap is a clean natural experiment on how a likelihood-trained decoder allocates a fixed budget under distributional pressure from the history.
Start from a failure bucket, then read history, beam and the judge's diagnosis side by side. This is where a reward-design idea gets sanity-checked against real rows.