Add some classification and ZeroShot classification tasks

[imenelydiaker]

Added these tasks with their zero shot version from CLIP benchmark:

• PatchCamelyon
• Country211
• GTSRB
• UCF101
• Imagenet1k

For the zero shot templates, I copied the ones from the CLIP Benchmark (found in datasets repositories), but we can agree on keeping only one prompt template per task for the evaluation.

I couldn't run UCF101 and Imagenet1k tasks (hardware issues), can someone help me run them please ?

Note: I ran these tasks on 1 experiment out of 5, just to confirm the task runs.

[KennethEnevoldsen]

Just to get into the new code I just added a PR here. It seems like the TaskMetadata updated from Main hasn't been carried over (or has accidentally been re-added).

[KennethEnevoldsen]

is this required? It should probably be conditional on the "modality"

[imenelydiaker]

Yeah I'm not sure avg_character_length is required, although it was filled for other image tasks. cc @isaac-chung did you update the metadata?

[isaac-chung]

I didn't. If it can be removed without triggering any pydantic errors, let's remove it for pure image datasets.

[KennethEnevoldsen]

What is a Scene? (should we add a descript of the domains somewhere?)

[imenelydiaker]

Yeah we definetly should add a description of domains. I copied "Scene" from other tasks as it looked more appropriate for these tasks

[KennethEnevoldsen]

This one should have been deleted from main

[isaac-chung]

Just to get into the new code I just added a PR here. It seems like the TaskMetadata updated from Main hasn't been carried over (or has accidentally been re-added).

I've added those changes in my previous PRs to the mieb branch. Just wondering if @imenelydiaker your base branch is up to date. Might just need a pull. Though I might have missed some changes as well (e.g. not removing all the old fields)

[isaac-chung]

I couldn't run UCF101 and Imagenet1k tasks (hardware issues), can someone help me run them please ?

Maybe we could run a downsampled version with 1 experiment run first? Then whoever has access to a cluster can run the full version?

[imenelydiaker]

Just to get into the new code I just added a PR here. It seems like the TaskMetadata updated from Main hasn't been carried over (or has accidentally been re-added).

I've added those changes in my previous PRs to the mieb branch. Just wondering if @imenelydiaker your base branch is up to date. Might just need a pull. Though I might have missed some changes as well (e.g. not removing all the old fields)

I pulled mieb and merged it into my branch earlier today (at 11:26 am). The only changes I made were adding new fields to Domains list, I guess it's not on my end..

[gowitheflow-1998]

@imenelydiaker thanks for adding these! The zero-shot results look much lower than CLIP paper (see page 43). I think it's because index gets wrong after combing prompt list (label index not correspondig to prompt text index). Can you test if the results are correct with one prompt first, and leave other template commented there? I think we need to modify the evaluator a bit to make it work but could be done in the next commit!

Thanks for raising combining prompts though. CLIP paper did the same. I think this is something worth adding. To make it work, we need to change the label index in the evaluator. e.g., if label index was 0,1,2, with 3 prompt templates structured in the way you do now, the label will become [[0,1,2], [3,4,5], [6,7,8]]. For an example with groundtruth 0, two ways we can implement it: 1. as long as long it hits one of 0,1,2, it is marked as correct for that example. 2. Ensemble probabilities of 0,1,2, and see if it's higher than ensemble of other label candidates.

For those that couldn't be run, might be great to change to 1 run instead of 5 like @isaac-chung suggested on the weekend, but keeping the full set (avoid remaking the full set at the end). We can run them at once at the end.

[gowitheflow-1998]

For images, we might need to change the way to process inputs and pass them into Evaluators. e.g.,

for t2i retrieval, it's now

queries = {query["id"]: {"text": query["text"]} for query in queries}
corpus = {image["id"]: image["image"] for image in corpus}
self.corpus[split], self.queries[split], self.relevant_docs[split] = (
    corpus,
    queries,
    qrels,
)

For zero-shot and linear probing ones, we have:

evaluator = ZeroshotClassificationEvaluator(
    dataset[self.image_column_name],
    dataset[self.label_column_name],
    candidate_labels,
    task_name=self.metadata.name,
    **kwargs,
)

X_sampled, y_sampled, idxs = self._undersample_data(
    train_split[self.image_column_name],  # type: ignore
    train_split[self.label_column_name],  # type: ignore
    self.samples_per_label,
    idxs,
)

operations like corpus = {image["id"]: image["image"] for image in corpus} ``dataset[self.image_column_name], train_split[self.image_column_name] are taking huge memory. I will fix these soon with iterable operations like dataloaders.

[imenelydiaker]

@imenelydiaker thanks for adding these! The zero-shot results look much lower than CLIP paper (see page 43). I think it's because index gets wrong after combing prompt list (label index not correspondig to prompt text index). Can you test if the results are correct with one prompt first, and leave other template commented there? I think we need to modify the evaluator a bit to make it work but could be done in the next commit!

Thanks for raising combining prompts though. CLIP paper did the same. I think this is something worth adding. To make it work, we need to change the label index in the evaluator. e.g., if label index was 0,1,2, with 3 prompt templates structured in the way you do now, the label will become [[0,1,2], [3,4,5], [6,7,8]]. For an example with groundtruth 0, two ways we can implement it: 1. as long as long it hits one of 0,1,2, it is marked as correct for that example. 2. Ensemble probabilities of 0,1,2, and see if it's higher than ensemble of other label candidates.

For those that couldn't be run, might be great to change to 1 run instead of 5 like @isaac-chung suggested on the weekend, but keeping the full set (avoid remaking the full set at the end). We can run them at once at the end.

So I managed to run everything with 1 experiment only. For ZeroShot, I put back 1 prompt only, handling multiple prompts requires more changes than what I expected and may affect the text-only version of MTEB.

If everything is good here, this PR should be ready to be merged.

[imenelydiaker]

Tests are failing because some dependencies are missing for FlagEmbedding, we may need to add them on mieb first and then we'll merge it into this branch.

cc @gowitheflow-1998

[gowitheflow-1998]

thanks! The results look great now. I will merge it and take care of the dependency issue. Will add implementations for multiple prompts at some point at well.