Finetune the released MetaCLIP models on my own dataset

[ShuxunoO]

Hello~

How can I finetune the released MetaCLIP models on my own dataset(Img-Txt paires) and how do I organize the data?

[howardhsu]

Thx, for your interests in this project. It can be very similar to pre-training and here's a suggested steps:

1. check the pre-training setup: https://github.com/facebookresearch/MetaCLIP#training
2. derive a new config from b32_400m and add `pretrained="metaclip_400m"

def b32_400m_ft():
    return b32_400m(
        pretrained="metaclip_400m",  # should download pre-trained MetaCLIP ViT-B/32 400M
    )

(we will have a new config system released soon)

3. depends on your fine-tuning set format, properly set train_data for https://github.com/facebookresearch/MetaCLIP/blob/main/src/training/data.py and we highly recommend use your own dataloader to control randomness instead of built-in one.