[MIEB] Add image classification and zero shot classification tasks

[isaac-chung]

First batch of image classification datasets, which covers

• Caltech101
  • Linear probe
  • zero shot
• CIFAR10
  • Linear probe
  • zero shot
• CIFAR100
  • Linear probe
  • zero shot
• FGVCAircraft
  • Linear probe
  • zero shot
• Food101
  • Linear probe
  • zero shot
• OxfordPets
  • Linear probe
  • zero shot

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

Adding datasets checklist

Reason for dataset addition: ...

• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
  • mteb run -m openai/clip-vit-base-patch32 --model_revision 3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268 --output_folder results-mieb -t Caltech101
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[gowitheflow-1998]

thanks! amazing speed. I think for these ones, we would like to implement evaluation with both AbsTaskImageClassification (linear probe) and also zero-shot ones with AbsTaskZeroshotClassification. Text candidate prompts for zeroshot can be found
p42 here in the CLIP paper, e.g., "a photo of kangaroo" and the rest of 101 for Caltech101. cc @imenelydiaker for information as well.

Also why is test failing for loading caltech101?

[isaac-chung]

I think for these ones, we would like to implement evaluation with both AbsTaskImageClassification (linear probe) and also zero-shot ones with AbsTaskZeroshotClassification.

Ooooh no problem. I was confused by the sheet where it said I; or I-T zero shot - thought it meant either-or, not both. I can add the zero-shot version as well though 👍

Also why is test failing for loading caltech101?

I am also perplexed by this. The link we use to test works. [edit] think I found a fix. Pushed.

[isaac-chung]

For CIFAR, the implementation here is 9-20 points short compared to the CLIP paper for both linear probing and zero shot on the B/32 model 🤔 Same for Caltech101. Resolved!