1.21.4

1.21.4 (2024-12-04)

Documentation

• docs: Add Model Meta parameters and metadata (#1536)

• add multi_qa_MiniLM_L6_cos_v1 model meta

• add all_mpnet_base_v2

• add parameters to model meta

• make lint

• add extra params to meta (5fa7b7b)

Fix

• fix: add more model meta (jina, e5) (#1537)

• add e5 model meta

• address review comments (36bab4d)

1.21.3

1.21.3 (2024-12-02)

Fix

• fix: Proprietary models now get correctly shown in leaderboard (#1530)

• Fixed showing proprietary models in leaderboard

• Added links to all OpenAI models

• Fixed table formatting issues

• Bumped Gradio version (39349ff)

1.21.2

1.21.2 (2024-12-01)

Fix

• fix: Task load data error for SICK-BR-STS and XStance (#1534)

• fix task load data for two tasks

• correct dataset keys (5b6f20f)

1.21.1

1.21.1 (2024-11-30)

Fix

• fix: Correct typos superseeded -> superseded (#1532)

fix typo -> superseded (343b6e0)

1.21.0

1.21.0 (2024-11-29)

Feature

• feat: Evaluate missing splits (#1525)

• fix: evaluate missing splits (#1268)

• implement partial evaluation for missing splits

• lint

• requested changes done from scratch

• test for missing split evaluation added

• uncomment test

• lint

• avoid circular import

• use TaskResult

• skip tests for now

---

Co-authored-by: Isaac Chung <[email protected]>

• got test_all_splits_evaluated passing

• tests passing

• address review comments

• make lint

• handle None cases for kg_co2_emissions

• use new results info

---

Co-authored-by: Thivyanth <[email protected]> (8e12250)

1.20.6

1.20.6 (2024-11-29)

Fix

• fix: Adding missing metadata on models and mathcing names up with the results repo (#1528)

• Added Voyage 3 models

• Added correct metadata to Cohere models and matched names with the results repo (b02ae82)

1.20.5

1.20.5 (2024-11-29)

Documentation

• docs: Add lang family mapping and map to task table (#1486)

• add lang family mapping and map to task table

• make lint

• add back some unclassified lang codes (cfd43ac)

Fix

• fix: Ensure that models match the names on embedding-benchmarks/results (#1519) (e3d2b54)

Unknown

• Update tasks table (377a63d)

1.20.4

1.20.4 (2024-11-27)

Fix

• fix: align readme with current mteb (#1493)

• align readme with current mteb

• align with mieb branch

• fix test (942f212)

1.20.3

1.20.3 (2024-11-27)

Fix

• fix: leaderboard only shows models that have ModelMeta (#1508)

Filtering for models that have metadata (35245d3)

1.20.2

1.20.2 (2024-11-27)

Fix

• fix: Leaderboard demo data loading (#1507)

• Made get_scores error tolerant

• Added join_revisions, made get_scores failsafe

• Fetching metadata fixed fr HF models

• Added failsafe metadata fetching to leaderboard code

• Added revision joining to leaderboard app

• fix

• Only show models that have metadata, when filter_models is called

• Ran linting (0affa31)