Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Models refactoring, add stella, BGE, Linq #1225

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

AlexeyVatolin
Copy link
Contributor

@AlexeyVatolin AlexeyVatolin commented Sep 18, 2024

I created the SentenceTransformerWrapper class to remove code duplication and make it easier to add new models. I also added the task_type parameter to the model, as it is needed for some instructions.
Added models Stella 400M, 1.5B, Linq-Embed-Mistral and bge-multilingual-gemma2.

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Adding datasets checklist

Reason for dataset addition: ...

  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
    • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
    • intfloat/multilingual-e5-small
  • I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
  • If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
  • I have filled out the metadata object in the dataset file (find documentation on it here).
  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Adding a model checklist

  • I have filled out the ModelMeta object to the extent possible
  • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision_id) and
    • mteb.get_model_meta(model_name, revision_id)
  • I have tested the implementation works on a representative set of tasks.

@Samoed
Copy link
Contributor

Samoed commented Sep 18, 2024

I think that this PR very similar to #1221

@KennethEnevoldsen
Copy link
Contributor

Thanks @AlexeyVatolin, sorry for the slow reply on this - as @Samoed says it is very similar to #1221 (though I think they are compatible). Had hoped to implement some of these changes in SentenceTransformers. Will finish off #1221 as most of the debates are already there, but I will get back to this PR after

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants