Add datasets for a benchmark newly introduced for "Engineering" domain #1911
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Added "Engineering" as new domain to TaskMetadata.py * Updated tasks table in docs * Updated task metadata for BuiltBenchClustering S2S and P2P
Samoed
reviewed
Jan 30, 2025
There was a problem hiding this comment.
It seems you want to add missing tasks to benchmark. You can follow adding_a_benchmark doc to fully integrate it
|
Makes perfect sense; I'll be adding other tasks and putting it all together as a new benchmark |
|
Just converted this to draft. Feel free to mark it as ready when it's ready for review :) |
|
Awesome! will do it shortly; tnx for your time! |
- Add BuiltBenchRetrieval task - Add BuiltBenchReranking task - Update metadata for BuiltBenchClusterinP2P - Update metadata for BuiltBenchClusterinS2S
mehrzadshm
changed the title
mehrzadshm
added a commit
to mehrzadshm/results
that referenced
this pull request
Feb 6, 2025
* add initial results for proposed tasks * update paths.json
2 tasks
|
Just followed the advice and integrated all changes, including new datasets and an associated new benchmark class in mteb, plus adding results in embeddings-benchmark/results (related PR: embeddings-benchmark/results#110) Just marking the PR ready for review :) Looking forward to your feedback. Thanks in advance for your time! |
Samoed
approved these changes
Feb 6, 2025
Samoed
requested review from
x-tabdeveloping,
KennethEnevoldsen and
isaac-chung
Co-authored-by: Roman Solomatin <[email protected]>
Co-authored-by: Roman Solomatin <[email protected]>
Co-authored-by: Roman Solomatin <[email protected]>
Samoed
pushed a commit
to embeddings-benchmark/results
that referenced
this pull request
Feb 7, 2025
…110) * Add BuiltBench results (related mteb PR: embeddings-benchmark/mteb#1911) * add initial results for proposed tasks * update paths.json * Update model_meta files modified in BuiltBench PR: #110 * rollback paths.json (see PR: #110)
isaac-chung
approved these changes
Feb 8, 2025
| "BuiltBenchReranking", | ||
| ], | ||
| ), | ||
| description="\"Built-Bench\" is an ongoing effort aimed at evaluating text embedding models in the context of buit asset management, spanning over various dicsiplines such as architeture, engineering, constrcution, and operations management of the built environment.", |
There was a problem hiding this comment.
Tiny typo
Suggested change
| description="\"Built-Bench\" is an ongoing effort aimed at evaluating text embedding models in the context of buit asset management, spanning over various dicsiplines such as architeture, engineering, constrcution, and operations management of the built environment.", | |
| description="\"Built-Bench\" is an ongoing effort aimed at evaluating text embedding models in the context of built asset management, spanning over various dicsiplines such as architeture, engineering, constrcution, and operations management of the built environment.", |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The datasets are associated with this research work (under review): Benchmarking pre-trained text embedding models in aligning built asset information
The initial results are included in embeddings-benchmark/results; related PR: #110
The proposed benchmark introduces 4 tasks, under three main types: clustering, retrieval, and reranking.
HuggingFace links to datasets :
Adding datasets checklist
Reason for dataset addition: This dataset points to a new domain, i.e., "Engineering" (more specifically related to architecture, construction, and built asset management)
mteb -m {model_name} -t {task_name}command.sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2intfloat/multilingual-e5-smallself.stratified_subsampling() under dataset_transform()make test.make lint.Looking forward to your feedback!