1.15.2

1.15.2 (2024-10-03)

Fix

• fix: derive results_directory path from results_repo name (#1275)

fix: don't hardcode repo name when downloading results (b589c29)

• fix: Select benchmarks CLI option (#1261)

• add test case for a list of Benchmarks

• add selecting benchmarks CLI option

• typos

• use a separate attribute for benchmarks

• try fixing tests

• should accept string as well

• revert filename change

• use Benchmark and avoid circular import (e717d6e)

1.15.1

1.15.1 (2024-10-03)

Fix

• fix: Add Touche2020v3 and JMTEB (#1262)

• add datasets

• fix metrics

• add Touche2020v3

• fix metadata

• Apply suggestions from code review

Co-authored-by: Kenneth Enevoldsen <[email protected]>

• upd name and supress

• add benchmark class

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (5074918)

Unknown

• Update tasks table (7a55c85)

1.15.0

1.15.0 (2024-10-03)

Documentation

• docs: Update mteb(eng) calculation (#1258)

• Update mteb(eng) calculation

• Fixed citations

• Update MTEB(eng) + MTEB(multilingual) (11518ed)

• docs: Update affiliation (#1248)

• Update points.md

• Update points.md

---

Co-authored-by: Kenneth Enevoldsen <[email protected]> (647c295)

Feature

• feat: leverage SentenceTransformers' query/passage specific prompts (#1221)

• feat: leverage SentenceTransformer models' query/passage specific prompts

• refactor: remove E5Wrapper

fix: wrong e5 revisions

• fix: default prompt_type to None

• fix: e4ce987 revision no longer exists for multilingual-e5-small on the Hub

• fix: keep prompt_name in kwargs when model doesn't have a prompts attr

• feat: use Enum for prompt_type

• docs: specify how to use prompts with Sentence Transformers

• feat: readd arctic models due to metadata (c809b84)

1.14.26

1.14.26 (2024-09-29)

Fix

• fix: Add listing all available benchmarks CLI option (#1256)

• add benchmarks.md in README

• add cli option

• add benchmark cli test case

• correct typo (5e1e290)

1.14.25

1.14.25 (2024-09-29)

Fix

• fix: Get meta from CrossEncoder (#1255)

• remove indent after return

• handle cross encoders for model meta

• make lint

• update filename since we now have model name (0ad5dad)

1.14.24

1.14.24 (2024-09-28)

Documentation

• docs: Small point changes & more contributors (#1254)

• Update points.md

• Fix format

• Fix attribution (0d7664d)

Fix

• fix: Downsample large retrieval datasets (#1236)

• most tasks

• lint

• fix other issues

• refactor

• lint and docs

• add polish

• keep case sensitive mteb paths

• add potential points

• fix points

• fix test about metadata

• update tasks and stats

• lint (b754f1a)

Unknown

• Update tasks table (e875401)

• Update points table (a155401)

• Update points table (55e18a8)

1.14.23

1.14.23 (2024-09-28)

Documentation

• docs: Added coordination point for Jimmy Lee (#1253)

docs: Added coordination point for Jimmy lee for his work on the coordination of Crystina and Nandan (6b27ce0)

• docs: Update affiliation (#1247)

Update points.md (45de3ec)

Fix

• fix: Add multilingual Benchmark (#1252)

• fix: Add multilingual bench

• Update mteb/benchmarks/benchmarks.py

Co-authored-by: Niklas Muennighoff <[email protected]>

• format

---

Co-authored-by: Niklas Muennighoff <[email protected]> (6a6259c)

Unknown

• Update points table (4dbf784)

• Update points table (35abd84)

• Add final author list (9ac24b7)

• Merge branch 'main' of https://github.com/embeddings-benchmark/mteb (8582244)

• Added author-information (c02c8e5)

1.14.22

1.14.22 (2024-09-27)

Documentation

• docs: Create benchmarks overview table (#1245)

• fix get_benchmarks method

• add create benchmark script

• make lint (fda9be1)

• docs: Add MTEB(code) dataset (#1237)

• docs: Add MTEB(code) dataset

• Fix linting (f808863)

• docs: Update points (#1228)

• Fix case

• Fix casing

• Fix case

• Fix case

• Create 971.jsonl

• Update contrib

• Add contributors (a636dc2)

Fix

• fix: @mrshu's name in points.md (#1246)

• Use the diacritic character to be inline with Slovak spelling.

Signed-off-by: mr.Shu <[email protected]> (3c06694)

Unknown

• Add contributor (#1243) (18f7306)

• Update of my affiliation (#1242)

Update points.md (7469afa)

• Update points table (eece6ec)

• Update points table (dddf5c2)

1.14.21

1.14.21 (2024-09-20)

Documentation

• docs: clarify adding a model (#1222) (25b7a2f)

Fix

• fix: Add RepLLaMA style models (#1223)

• init commit

• working and reproducing

• lint

• update hashes

• warning

• add pyproject (bedcfb3)

Unknown

• Update points table (694e563)

• Update points table (4de39ee)

• Update points table (b66aaeb)

1.14.20

1.14.20 (2024-09-17)

Fix

• fix: Allow benchmark to specify eval_splits (#1217)

• fix: Allow benchmark to specify eval_splits

This PR allow for benchmarks to specify specific eval. splits. This allow us to fully specify a benchmark within the benchmark object.

To do this it add the following:

• added eval_splits to the Abstask object, which default to metadata.eval_splits
• use the task.eval_splits unless overwritten in mteb.MTEB.run
• added eval_splits arg to mteb.get_tasks, which filter the tasks based on splits
• updated documentation
  • renamed the "Advanced Usage" to "Usage Documentation" to make it more accicible
• added tests where relevant

• Added correction based on feedback (00260b5)