MiniOneRec 0.5B Reproduction & GRPO Reward Extension

Generative Recommendation · Semantic ID · Qwen2.5 · SFT · GRPO

This repository is a reproduction and experimental extension of AkaliKong/MiniOneRec. It packages a single-RTX-4090 FAST pipeline and a controlled reward-ablation study; it does not claim the original MiniOneRec framework as my own work.

Original project: https://github.com/AkaliKong/MiniOneRec

Overview

This fork builds on MiniOneRec's open-source generative-recommendation workflow: RQ-VAE semantic IDs, Qwen2.5-0.5B supervised fine-tuning, constrained beam-search evaluation, and Group Relative Policy Optimization (GRPO). The experimental extension is a SID-aware Target-Relative Hard Negative Reward that uses the hierarchy already encoded in item SIDs.

What I Added

1. Built a reproducible 0.5B FAST pipeline for one NVIDIA RTX 4090.
2. Created deterministic train and test subsets with fixed manifests and seed=42.
3. Exposed controls such as max_steps for short SFT and GRPO experiments.
4. Fixed the small-data GRPO sample(n=10000) crash.
5. Implemented the SID-aware Target-Relative Hard Negative Reward.
6. Added CPU unit tests for reward behavior, validation, weighting, and compatibility.
7. Ran a matched Original/Improved GRPO controlled experiment from the same SFT checkpoint.
8. Added result collection, fixed-scope evaluation, and experiment-integrity checks.

Method

MiniOneRec's original recommendation reward combines rule correctness and a rank-aware term:

R_original = R_rule + R_rank

This fork adds a target-relative penalty only for an incorrect candidate. Let LCP be the continuous common-prefix length between the candidate SID and target SID across the three-level hierarchy:

R_target_relative = -lambda_sid * LCP(candidate_sid, target_sid) / 3
R_total = R_rule + R_rank + R_target_relative
lambda_sid = 0.2

The term is target-relative: a wrong item sharing the first SID level is treated as a harder negative, while sharing the first two levels is harder still. It requires no embedding lookup, extra model, or extra forward pass. The recommended CLI is:

--reward_type sid_target_relative \
--sid_reward_weight 0.2

sid_hard_negative remains accepted only as a backward-compatible alias. See SID_HARD_NEGATIVE_REWARD.md for implementation details.

Experimental Setup

Setting	Value
Dataset	Amazon Office Products
FAST train	3,892 fixed rows, seed=42
FAST evaluation	1,000 fixed examples, seed=42
Base model	Qwen/Qwen2.5-0.5B
GPU	NVIDIA RTX 4090
SFT	200 steps
GRPO	40 steps per arm

This is not the full Office benchmark. Original and Improved GRPO start independently from the same SFT checkpoint and use the same data, seed, training budget, batch configuration, learning rate, beta, rollout beam, and evaluation definition. Only the reward differs.

Results

Model	HR@5	HR@10	NDCG@5	NDCG@10
SFT	0.035	0.055	0.02429797	0.03079225
GRPO Original	0.036	0.055	0.02464099	0.03077317
GRPO + SID Target-Relative	0.036	0.057	0.02468482	0.03148857

Compared with Original GRPO, the SID-aware target-relative reward produced:

• HR@10: +3.64%
• NDCG@10: +2.32%

Original GRPO itself did not improve HR@10 over SFT and showed a very small decrease in NDCG@10 under this short-budget FAST setting. These results therefore should not be described as “GRPO significantly improves SFT.” Raw reported values and the integrity record are in results/fast_office_1k/.

Efficiency

Stage	Budget	End-to-end / wall time	Peak VRAM
SFT	200 steps	~339 s	~6.2 GB
GRPO Original	40 steps	~146 s	~13.6 GB
GRPO Improved	40 steps	~154 s	~13.6 GB

Reproduction

The commands below reproduce the fixed FAST protocol without downloading a checkpoint from this fork.

1. Environment

conda create -n minionerec-fast python=3.11 -y
conda activate minionerec-fast
pip install -r requirements.txt

2. Data

Prepare the upstream MiniOneRec Amazon Office Products data, SID index, and item metadata at the paths expected by gpu_scripts/. The fixed train/test CSV files and their manifests are versioned under experiments/data/. Validate all required inputs before training:

bash gpu_scripts/00_gpu_preflight.sh

3. SFT and evaluation

SFT_MAX_STEPS=200 bash gpu_scripts/01_fast_sft.sh
bash gpu_scripts/02_eval_sft.sh

4. Matched GRPO arms

GRPO_MAX_STEPS=40 bash gpu_scripts/03_grpo_original.sh
GRPO_MAX_STEPS=40 bash gpu_scripts/04_grpo_improved.sh
bash gpu_scripts/05_eval_grpo_original.sh
bash gpu_scripts/06_eval_grpo_improved.sh

5. Collect results

bash gpu_scripts/07_collect_results.sh

The scripts refuse to overwrite formal output directories. Keep the shared configuration fixed across the two GRPO arms.

Repository Structure

Path	Purpose
rl.py	GRPO training and reward selection
sft.py	SFT training with FAST-step controls
tests/	CPU reward and provider tests
gpu_scripts/	Preflight, training, evaluation, and result collection
experiments/data/	Deterministic subset files, generators, and manifests
results/fast_office_1k/	Small final report, aggregate metrics, and runtime patch
SID_HARD_NEGATIVE_REWARD.md	Formal reward design and compatibility notes

Limitations

• Evaluation uses a fixed 1,000-example FAST subset, not the full 4,866-example Office benchmark.
• Only one seed (42) was evaluated.
• SFT and GRPO use short experimental budgets.
• Results should be treated as a controlled pilot experiment.
• Full-benchmark and multi-seed validation remain future work.

Acknowledgement

This work is based on the official MiniOneRec project. I thank the MiniOneRec authors for releasing the framework and preserving an open foundation for reproducible generative-recommendation research.

Original paper: MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation

@misc{MiniOneRec,
  title={MiniOneRec: An Open-Source Framework for Scaling Generative Recommendation},
  author={Xiaoyu Kong and Leheng Sheng and Junfei Tan and Yuxin Chen and Jiancan Wu and An Zhang and Xiang Wang and Xiangnan He},
  year={2025},
  eprint={2510.24431},
  archivePrefix={arXiv},
  primaryClass={cs.IR}
}

The original Apache-2.0 LICENSE is retained. Upstream authorship and attribution remain intact.