Ignore novel class areas in the semantic branch

[LiheYoung]

Hi,

I notice that you ignore all novel class areas (set their values as 255) in the semantic branch during training.

for i, query_label_tmp in enumerate(query_labels_tmp):
        query_label_tmp[labels[i] == 255] = 255
        for j in range(len(baseclass_ids)):
            if baseclass_ids[j] == 0:
                continue
            elif baseclass_ids[j] == 255 or baseclass_ids[j] not in label_sets[cfg['label_sets']]:
                query_label_tmp[labels[i] == baseclass_ids[j]] = 255
                continue
            query_label_tmp[labels[i] == baseclass_ids[j]] = baseclass[cfg['label_sets']][baseclass_ids[j]]

However, it is unrealistic for us to know which pixels belong to novel classes. It may be more realistic to set them as background, same as the common practice in meta-training process.

Is there anything I misunderstand? Thank you in advance!

[youngfly11]

Thanks for your remind. We will retrain our code to further check it. We will report it once we get more results.

[youngfly11]

Hi, Lihe:
We modified our code by setting the novel class to 0 in gt map. Here are the results.

Experiments

(In paper Tab1, ours w/o $s^u$ experiments, pascal 1way1shot )

Original (set 255): 47.83, 58.75, 53.80, 45.63, 51.50
Retrain (set 0): 48.12, 58.79, 52.82, 46.17, 51.48

We found that two strategies perform differently in splits. But on average, they have little difference. Our network will not learn the novel information by ignoring those classes.

[LiheYoung]

Thanks for your experiments, Yongfei!

My original concern on setting them as 0 is that features of some latent novel objects may be trained to be closer to true backgrounds, which will degrade the generalization ability during testing. And ignoring them may alleviate this problem.

Actually, I observe similar results in my experiments. The decrease in the 3rd split may result from the person class, since it appears frequently as background during training.