Research Insights for Generative Recommendation

After </think>, the training sampler can write any tokens. The reward code reads the first three SID tokens and adds 0.1 if they form a real catalog item. Evaluation is simpler: its prefix tree blocks every invalid SID before it can be generated.

Sample the CoT exactly as in the current baseline. After </think>, use the catalog trie and repeatedly choose the highest-scoring legal next token until one complete SID is produced. Each CoT still produces exactly one answer. Once every answer is valid by construction, remove the 0.1 validity bonus.

The CoT remains a stochastic rollout, but the SID is selected by deterministic search rather than sampled from the policy. The greedy SID tokens are therefore not ordinary on-policy GRPO actions. If they are included in the loss, describe the update as search-guided or reward-weighted training rather than claiming that group normalization or importance sampling makes constrained greedy search unbiased.

1. Compare the current unconstrained answer decoder plus validity bonus against one constrained greedy SID per sampled CoT, with no validity bonus and a matched rollout-token budget. Keep every other reward term unchanged.
2. Reuse the existing catalog prefix map, but add the legal Code-1 root set and test all boundary cases: malformed CoT ending, duplicated SID tokens, premature EOS and catalog items whose prefixes share tokens.
3. Log the baseline invalid-answer rate, probability mass removed by the mask, reward variance within each 16-rollout group and zero-advantage group rate.
4. Do not use candidate diversity as A1's success criterion. That question belongs to A2.

If invalid outputs are common, constrained generation should remove wasted rollouts and increase useful reward variation per generated token without lowering Hit@10/NDCG@10. If invalid outputs are already rare, remove the unnecessary bonus but do not expect this change alone to improve recommendation quality.

For every rollout, calculate_reward returns 0, 0.25, 0.5 or 1.0 from the candidate's shared SID prefix with the target, plus the validity bonus. GRPO subtracts the mean score of the 16 responses, so the final advantage is relative. But the raw reward for one candidate is unchanged whether the other 15 are duplicates, useful alternatives or completely unrelated.

For one prompt, sample K CoTs and stop each one at </think>. For every fixed CoT c_m, run beam search with width B to produce one ranked list S_m. Compute one overall score R_m for that B-item list. The GRPO group therefore contains K CoTs and K list scores—not K × B candidate rollouts.

Normalize the K list scores for the same prompt: A_m = (R_m − mean(R_1…R_K)) / (std(R_1…R_K) + ε). Apply A_m once to the sampled tokens of CoT c_m. The B beam candidates are used only to evaluate that CoT; they do not receive separate GRPO advantages.

The stochastic GRPO actions are the K sampled CoTs. Beam search is a deterministic procedure that maps each fixed CoT to a ranked item list. Because beam tokens are not included in the policy loss, they do not need candidate-level leave-one-out rewards or on-policy log-probability correction.

1. In ray_trainer.py, generate K CoTs for each prompt and stop them at </think>. Keep the current prompt-level uid for GRPO grouping and give every CoT its own cot_uid.
2. For each original prompt + frozen CoT, call the existing beam helper in vllm_rollout_spmd.py with beam width B. Return the ranked SID list and beam scores as metadata rather than as B training responses.
3. Extend BatchRewardManager to compute one list reward from each beam and return exactly K scalar rewards. Keep only the sampled CoT tokens in the actor loss mask.

Use three terms on the same 0–1 scale: R(S_m) = α · ExactNDCG@B + β · SoftRel@B − γ · Redundancy@B, with α largest so exact target ranking remains the primary objective. Rank the B candidates in their beam order before computing every term.

1. ExactNDCG@B. There is one ground-truth SID. If its first occurrence is at rank r, the score is 1 / log2(r + 1); if it is absent, the score is 0. Earlier exact hits receive more reward.
2. SoftRel@B. Give each candidate a relevance score g_i ∈ [0,1]: exact match is 1, sensible alternatives receive partial credit, and unrelated items receive 0. Discount lower ranks and take the weighted average Σ_i g_i / log2(i+1) ÷ Σ_i 1 / log2(i+1). The denominator is fixed, so an all-bad list remains near 0.
3. Redundancy@B. Among candidates with g_i ≥ τ, measure pairwise collisions at the full-SID, Code-1+2 and Code-1 levels, then average them: Σ_b n_b(n_b−1) / [n(n−1)]. A score of 0 means no collisions; 1 means all candidates collapse to the same item or branch. Use 0 when fewer than two candidates pass the threshold.

For a pilot, give one frozen LLM the interaction history, ground-truth target and the entire B-item beam in a single call. Ask it to assign anchored 0–1 scores for target-need satisfaction: exact target is fixed at 1, a genuine substitute can receive high partial credit, a merely related item receives less, and an unrelated item receives 0. Use those values to compute SoftRel@B. Do not use a purely relative ranking: when all candidates are bad, the least bad one still needs an absolute score near zero.

1. Use the online LLM only for a small controlled pilot. For a real run, collect teacher labels on training beams, then distil them into a frozen local reward model or a metadata/embedding scorer. This keeps cost and reward drift out of the RL loop.
2. Compare the current K complete CoT-item rollouts with K sampled CoTs scored by their B-item beams. Keep K fixed and also report a token-matched baseline because beam search adds decoding cost.
3. Ablate the list score in stages: ExactNDCG only; ExactNDCG plus SoftRel; and the full score with Redundancy. This shows which term creates the gain.
4. Report reward variance and zero-advantage rate across the K CoTs, beam branch coverage, Hit@B and NDCG@B. Audit a sample of teacher scores for agreement and popularity bias.

CoTs that produce stronger beams should receive higher GRPO advantages, and the final beam should improve target-branch coverage or exact ranking metrics. If the K CoTs usually receive the same list score, this setup provides too little comparative signal.

In 72%–81% of misses, no beam item has the target's Code-1 prefix. The earlier 'interest coverage gap' analysis also shows that mentioning an interest is not the same as selecting it or assigning it a beam slot.

Start from the original baseline rather than from A1 or A2. After the CoT, require the model to output a short ranked list of structured interests before it outputs an item SID. Train that ranking against the target category or SID prefix. During evaluation, allocate the existing ten slots across the selected interests. Do not add A2's fixed-CoT list reward; A3 asks only whether explicit interest selection improves branch coverage.

1. Use a 2×2 ablation: intent supervision off/on and standard beam versus intent-aware slot allocation. This separates a better policy from a better decoder.
2. Measure target-intent Recall@M, target Code-1 and Code-1+2 coverage@10, final Hit@10/NDCG@10, and relevance of the extra branches.
3. If target-branch coverage rises but the exact target still misses, inspect within-branch ranking rather than adding still more diversity.

The target branch should appear more often without filling the list with irrelevant branches. The expected sequence is simple: the model selects the right interest more often, the beam covers that branch more often, and exact ranking then improves.

Across the three domains, 23%–35% of examples fall into the two lowest levels on a five-level predictability scale. These labels mean the judge found little or no evidence for the target in the observed history. In Video Games, those 1,398 examples produce only 7 of the current baseline's 616 hits.

Early training should make faster progress when most sampled examples can produce different rewards across their rollouts. Harder examples can be added later, after the policy has learned enough to reach their target branches more often.

The predictability model and the candidate reward model may share an encoder, but they answer different questions and need separate heads. The predictability head sees (history, target) and estimates how much evidence the history contains for that target; it controls sampling weight or rollout budget. The reward head sees (history, target, candidate) and estimates whether that particular candidate is a sensible non-exact recommendation. A predictable target does not make an arbitrary candidate good, so the predictability score itself must not be reused as the item reward.

1. Label a training subset with the same predictability rubric. For scale, fit a lightweight difficulty model using available history/target features such as history length, SID-prefix concentration, repeat status and target-to-history relation. Do not rely on timestamps that are absent from the Phase-3 data.
2. Compare four samplers at matched rollout-token budget: uniform, hard filtering, soft predictability weighting, and a staged curriculum that begins with scores from levels 4 and 5 and then adds level 3. Keep a small sample from levels 1 and 2 throughout rather than assuming those cases are universally impossible.
3. Keep the evaluation split unchanged. Report Hit@10 and NDCG@10 for every predictability score, and separate exact repeats from genuinely new targets.

At the same rollout-token budget, training should produce more groups with useful reward differences and later improve level-3 and novel targets. If gains appear only on high-predictability exact repeats, the curriculum taught copying rather than broader preference.

In the judged snapshots, CoTs that identify the target interest have 2.4×–11.6× as high Hit@10. On Video Games, the gap is 13.6% versus 5.6%. This is correlation, but it identifies a promising process signal for a controlled reward experiment.

Let a frozen scorer compare the interest inferred by the reasoning with the interest behind the ground-truth item and output s_intent ∈ [0,1]. A clear match receives a score near 1, a related but incomplete interest receives partial credit, and a generic or wrong interest receives 0. Add it to the current item reward as R = min(1, R_item + λ_intent · s_intent), with λ_intent small enough that exact item match remains the strongest signal.

1. Compare the current SID-prefix reward, a binary target-interest reward, and the graded s_intent reward at matched compute.
2. Create offline labels with examples of exact interest alignment, partially correct interests, generic restatements and clearly wrong interests. Distil those labels into a frozen local scorer rather than calling an LLM inside every rollout.
3. Audit whether broad category words or copied target metadata receive undeserved credit. Semantic proximity should reflect the user's likely need, not surface word overlap.

The progressive reward should create useful differences between otherwise unsuccessful rollouts and eventually improve exact metrics, especially at predictability levels 3 and 4. If only the scorer's own alignment score improves while Hit@10 stays unchanged, better reasoning is not reaching candidate generation.

Among predictable misses, 79%–89% contain a substitute, same-category item or complement according to the judge. The existing prefix reward recognises only candidates encoded near the target in the SID tree; it cannot tell whether a candidate outside that branch is a useful alternative or merely unrelated.

A candidate makes sense when, given the user's history, it is a defensible next recommendation and captures the same interest or need as the ground-truth target. A genuine substitute, product variant or natural same-series continuation can qualify. Sharing only a broad category, brand or popularity prior is not enough.

Let a frozen reward model output p_sense = P(makes_sense | history, target, candidate). Use the piecewise item reward 1 for an exact SID match; λ · p_sense for a non-exact candidate only when p_sense ≥ τ; and 0 otherwise, with 0 < λ < 1. Exact match bypasses the learned model and can never be outscored. This p_sense is also the candidate gain g_i used by A2's SoftRel@B.

1. Use one LLM teacher call to label several candidates jointly for each training example. Include exact positives, judged sensible non-exact candidates, unrelated negatives, and hard negatives that share a category or brand but do not satisfy the same need.
2. Train a local reward model with a shared history/target encoder and a candidate scoring head. It may share the backbone with the R1 predictability model, but keep the two outputs and losses separate. Freeze and calibrate the reward model before RL.
3. Compare the current prefix ladder, the three-level reward above, and max(prefix_reward, λ · p_sense). Sweep λ and τ on held-out training data rather than the test split.
4. Audit precision among rewarded non-exact candidates. A high false-positive rate would let generic same-category items collect reward and displace exact targets.

The extra reward should create useful differences between non-exact candidates and improve final Hit@10/NDCG@10 while exact match remains dominant. If only the reward model's own score improves, lower λ, raise τ, or stop using that score as a training objective.