Skip to content

Commit

Permalink
update README
Browse files Browse the repository at this point in the history
  • Loading branch information
ArvinZhuang committed Oct 18, 2024
1 parent 0cbb080 commit ae1faa2
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 3 deletions.
15 changes: 12 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,16 @@
# Starbucks
Starbucks: Improved Training for 2D Matryoshka Embeddings
<p align="center">
<img src="Starbucks.png">
</p>

### General guidelines
We propose Starbucks: a new 2D MRL fine-tuning and pre-training method.

Starbucks is composed of two key processes: the Starbucks Masked Autoencoding (SMAE) pretraining and the Starbucks Representation Learning (SRL) fine-tuning processes.

In Starbucks, the model loss is computed based on a limited target list of layer-dimension pairs, ranging from smaller to larger sizes, much like how the coffeehouses chain [Starbucks](https://en.wikipedia.org/wiki/Starbucks) offers coffee in different cup sizes, from Demi to Trenta.

## General guidelines
Our codebase is built on top of torch and transformers.

We recommend using a conda environment to install the required dependencies.
Expand All @@ -22,12 +31,12 @@ For SRL fine-tuning on retrieval task, see [retrieval](retrieval/README.md).

For SRL fine-tuning on STS task, see [sts](sts/README.md).

### Model Checkpoints
## Model Checkpoints

We released our model checkpoints on Hugging Face Model Hub:

Pre-trained SMAE: [bert-base-uncased-fineweb100bt-smae](https://huggingface.co/ielabgroup/bert-base-uncased-fineweb100bt-smae)

Fine-tuned Starbucks_STS: [Starbucks_STS](https://huggingface.co/ielabgroup/Starbucks_STS)

Fine-tuned Starbucks_Retrieval: [Starbucks_Retrieval](https://huggingface.co/ielabgroup/Starbucks_Retrieval)
Fine-tuned Starbucks_Retrieval: [Starbucks-msmarco](https://huggingface.co/ielabgroup/Starbucks-msmarco)
Binary file added Starbucks.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit ae1faa2

Please sign in to comment.