[hieuddo]

Description

Many Semantic ID (SID) Recommendation models evaluate at SID level; if the model predicts the correct groundtruth SID, it's a perfect hit. However, one SID can map to multiple items, so predicting the correct SID does not necessarily identify the correct item; and it's also not fair for other item-based models (e.g., SASRec, BERT4Rec) in comparison.

This PR provides shared evaluation and resolution components for item-level evaluation for SID recommender models.

Implemented:

• Collision-Corrected Evaluation (CCE): computes expected item-level credit under uniform ordering within predicted SID buckets. Reference.
• Explicit item resolvers: training-popularity and history-cosine policies, plus custom scorer support for inference and evaluation.
• Adapt existing SID models (TIGER, LETTER, RPG, and DiffGRM), using item-level evaluation.

I added SID-item mapping/trace to cornac/models/semantic_id.py because it's not a Recommender abstract class, and to avoid making cornac/models/recommender.py bloated; not sure if it's appropriate.

Related Issues

Checklist:

• I have added tests.
• I have updated the documentation accordingly.
• I have updated examples/README.md (if you are adding a new example).

[saghiles]

Maybe better to move this out of models. It can go to utils for instance.

[hieuddo]

Sounds good. That'd be a better fit than inside models

[saghiles]

What is the difference between this file "semantic_id_evaluation.py" and "sid_evaluation.py"?

• I would recommend improving the naming. For instance, can we say that semantic_id_evaluation.py implements a NextItemEvaluation for Semantic-ID based recommenders? If yes we can name it explicitly
• Add more documentation explain why and when to use this eval over the base NextItem evaluation
• Are there any references in the literature supporting this method? If yes, we can point to them
• Is this method dedicated to SIDs based models are can be used to evaluate list of models including both SID-based (e.g., Tiger) and non-SIDs-based (e.g., Sasrec) models?
• Also: semantic_id_evaluation can be confusing, what we are evaluating here is recommendation performance, not the quality of semantic IDs.

[saghiles]

• Add more docs with a concrete example to explain what code inside this scripts allows to do.
• /eval_methods may not be the right path for this script. Consider moving it to utils.

[hieuddo]

@saghiles Thanks for the suggestions. I’ve consolidated the implementation into:

• cornac/eval_methods/semantic_id_evaluation.py: contains SemanticIDEvaluation(NextItemEvaluation), prediction collection, and metric computation from saved predictions. Most of the Semantic ID methods have been using leave-last-out evaluation method (which is the same as our NextItemEvaluation), but with these new changes in item-level evaluation, I think it'd be better to create a separate interface for Semantic ID.
• cornac/utils/semantic_id.py: contains shared item-to-SID mappings, SID predictions, prediction traces, and item resolvers.
• cornac/models/recommender.py: SemanticIDRecommender extends NextItemRecommender with interfaces for accessing item-to-SID mappings and raw SID predictions.

The documentation now distinguishes two ways to convert SID predictions into item-level results:

• CCE (one SID to all associated items): expands each predicted SID into its complete item bucket and computes expected metrics under uniformly random ordering within each bucket, following the CCE/ZCR paper.
• Explicit resolution (one SID to one item): selects a concrete member of each bucket.
  • Popularity selects the most frequent item in the training data
  • cosine selects the item most similar to the user’s training-history feature profile

[saghiles]

@saghiles Thanks for the suggestions. I’ve consolidated the implementation into:

• cornac/eval_methods/semantic_id_evaluation.py: contains SemanticIDEvaluation(NextItemEvaluation), prediction collection, and metric computation from saved predictions. Most of the Semantic ID methods have been using leave-last-out evaluation method (which is the same as our NextItemEvaluation), but with these new changes in item-level evaluation, I think it'd be better to create a separate interface for Semantic ID.
• cornac/utils/semantic_id.py: contains shared item-to-SID mappings, SID predictions, prediction traces, and item resolvers.
• cornac/models/recommender.py: SemanticIDRecommender extends NextItemRecommender with interfaces for accessing item-to-SID mappings and raw SID predictions.

The documentation now distinguishes two ways to convert SID predictions into item-level results:

• CCE (one SID to all associated items): expands each predicted SID into its complete item bucket and computes expected metrics under uniformly random ordering within each bucket, following the CCE/ZCR paper.

• Explicit resolution (one SID to one item): selects a concrete member of each bucket.

  • Popularity selects the most frequent item in the training data
  • cosine selects the item most similar to the user’s training-history feature profile

Hi @hieuddo , thank you so much for the clarification. to follow up,

1. From the description, this still seems to be a Next Item evaluation method. Ideally, the SID-to-ID mapping should be handled at the model level, with optional parameters to support different mapping strategies if needed. Evaluation methods should remain as model-agnostic as possible to preserve Cornac’s ability to compare a broad range of models. Their responsibilities should primarily be to prepare, sample, or split the data, obtain predictions from models, and evaluate those predictions. They should not dictate how models generate predictions or transform model outputs as part of model-specific logic.
2. Could we achieve the current objective without extending the existing NextItemEval method and enabling semantic-ID-based models, such as TIGER, to optionally perform various SID-to-ID mapping before passing their final predictions to the evaluation method?
3. Based on my current understanding, I would not recommend introducing a separate SemanticIDEval method, as it seems to overlap with the responsibilities of NextItemEval and could introduce unnecessary model-specific logic into the evaluation layer. Maybe let's get @qtuantruong input as well.

[hieuddo]

@saghiles Thanks for the suggestions. I’ve consolidated the implementation into:

• cornac/eval_methods/semantic_id_evaluation.py: contains SemanticIDEvaluation(NextItemEvaluation), prediction collection, and metric computation from saved predictions. Most of the Semantic ID methods have been using leave-last-out evaluation method (which is the same as our NextItemEvaluation), but with these new changes in item-level evaluation, I think it'd be better to create a separate interface for Semantic ID.
• cornac/utils/semantic_id.py: contains shared item-to-SID mappings, SID predictions, prediction traces, and item resolvers.
• cornac/models/recommender.py: SemanticIDRecommender extends NextItemRecommender with interfaces for accessing item-to-SID mappings and raw SID predictions.

The documentation now distinguishes two ways to convert SID predictions into item-level results:

• CCE (one SID to all associated items): expands each predicted SID into its complete item bucket and computes expected metrics under uniformly random ordering within each bucket, following the CCE/ZCR paper.

• Explicit resolution (one SID to one item): selects a concrete member of each bucket.

  • Popularity selects the most frequent item in the training data
  • cosine selects the item most similar to the user’s training-history feature profile

Hi @hieuddo , thank you so much for the clarification. to follow up,

1. From the description, this still seems to be a Next Item evaluation method. Ideally, the SID-to-ID mapping should be handled at the model level, with optional parameters to support different mapping strategies if needed. Evaluation methods should remain as model-agnostic as possible to preserve Cornac’s ability to compare a broad range of models. Their responsibilities should primarily be to prepare, sample, or split the data, obtain predictions from models, and evaluate those predictions. They should not dictate how models generate predictions or transform model outputs as part of model-specific logic.
2. Could we achieve the current objective without extending the existing NextItemEval method and enabling semantic-ID-based models, such as TIGER, to optionally perform various SID-to-ID mapping before passing their final predictions to the evaluation method?
3. Based on my current understanding, I would not recommend introducing a separate SemanticIDEval method, as it seems to overlap with the responsibilities of NextItemEval and could introduce unnecessary model-specific logic into the evaluation layer. Maybe let's get @qtuantruong input as well.

@saghiles, your design actually makes more sense. I tried to take on too big a scope for this PR.

Let me reduce this PR to SID-level vs. item-level evaluation metrics for NextItemEvaluation first (using CCE).

I'm thinking we can extend some flag for NextItemEvaluation for SID models; then enforce both SID- and item-level metrics (where applicable).

I'll skip the item resolvers for now and think it through first.

[qtuantruong]

I largely agree with @saghiles on trying to enable these SID models to work with NextItemEvaluation because that's still the final goal of a recommender. My suggestions are as follows:

1. Use a SemanticIDMixin to expose SID predictions and collision-group information. Keep explicit SID-to-item resolution policies in the model, different models might do it differently. Something like:

class SemanticIDMixin:
    def semantic_id_catalog(self):
        raise NotImplementedError

    def predict_semantic_ids(self, history_items, **context):
        raise NotImplementedError


class TIGER(NextItemRecommender, SemanticIDMixin):
    ...

2. Reuse NextItemEvaluation so SID models can be compared with other models such as SASRec, etc. That's still the goal I believe.
3. Add CCE as opt-in metrics, computed when the model exposes the required collision-group information. Unsupported models should still run their standard metrics; CCE results can be marked unavailable.

There will be other details to get right, but I hope we can align on the above to more forward.