Is it possible to calculate metric for groups seperately?

[bhuebner3]

Specifically, I would like to compute NDGC separately for different groups (in my case gender: male vs female). Please let me know if this is possible, or if there is a way to implement this?

[Wicknight]

Hello @bhuebner3 , I think this is feasible, and here is my solution :

1. Locate the NDCG class in RecBole/recbole/evaluator/metrics.py.
2. Then you can find the metric_info function in this file, just as follows:

    def metric_info(self, pos_index, pos_len):
        len_rank = np.full_like(pos_len, pos_index.shape[1])
        idcg_len = np.where(pos_len > len_rank, len_rank, pos_len)

        iranks = np.zeros_like(pos_index, dtype=np.float)
        iranks[:, :] = np.arange(1, pos_index.shape[1] + 1)
        idcg = np.cumsum(1.0 / np.log2(iranks + 1), axis=1)
        for row, idx in enumerate(idcg_len):
            idcg[row, idx:] = idcg[row, idx - 1]

        ranks = np.zeros_like(pos_index, dtype=np.float)
        ranks[:, :] = np.arange(1, pos_index.shape[1] + 1)
        dcg = 1.0 / np.log2(ranks + 1)
        dcg = np.cumsum(np.where(pos_index, dcg, 0), axis=1)

        result = dcg / idcg
        return result

The return value here is NDCG result for each user.
3. Get user groups from dataset.user_feat according to gender.
4. Index the results of step 2 based on the user group of step 3.

[bhuebner3]

Thanks for the help @Wicknight!
One last question, since there is no discussion board on https://github.com/TangJiakai/RecBole-FairRec
When running the FOCF model from FairRec, I run a baseline model with fair_objective = none and an optimized model with fair_objective = value, with all the other parameters being the same. The optimized model has slightly lower NDCG, and also higher Value Unfairness. It seems the point of the model is that the value unfairness should be decreased, but it has increased on all tests. Is there something I misunderstand?

[TangJiakai]

Thanks for the help @Wicknight! One last question, since there is no discussion board on https://github.com/TangJiakai/RecBole-FairRec When running the FOCF model from FairRec, I run a baseline model with fair_objective = none and an optimized model with fair_objective = value, with all the other parameters being the same. The optimized model has slightly lower NDCG, and also higher Value Unfairness. It seems the point of the model is that the value unfairness should be decreased, but it has increased on all tests. Is there something I misunderstand?

I have actually encountered the same problem, because the original problem was to solve the rating prediction, here we are taking implicit recommendation as the task, which is a clear difference. Secondly, even when converted to a rating prediction task, I tested it and actually had the same problem. I emailed the original author of the paper to consult him about this contradiction in the past, but unfortunately he did not give me a reply. Hope my answer could help you!