Loading private model with trust_remote_code=True fails

[deklanw]

st_model = SentenceTransformer(
    "borgcollectivegmbh/multitask_train_run_2",
    token=os.environ["BORG_HUGGINGFACE_TOKEN"],
    trust_remote_code=True,
)

The above fails with

OSError: borgcollectivegmbh/multitask_train_run_2 is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`

Despite my passing a token. I realized that by setting trust_remote_code=False it works, but my weights get reset. I realized even further that if the model is already cached then trust_remote_code=True works again. So, a horrible work-around for me is

for trust_remote_code in [False, True]:
      st_model = SentenceTransformer(
          "borgcollectivegmbh/multitask_train_run_2",
          token=os.environ["BORG_HUGGINGFACE_TOKEN"],
          trust_remote_code=trust_remote_code,
      )

where the first iteration caches it but resets the weights and the second iteration works properly.

Version:

    "sentence-transformers==3.3.1",

[tomaarsen]

Hello!

I was able to reproduce this with a clone of https://huggingface.co/nomic-ai/nomic-embed-text-v1.5. This is part of the traceback:

Traceback (most recent call last):
  File "c:\code\sentence-transformers\demo_3115.py", line 4, in <module>
    model = SentenceTransformer(
            ^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\SentenceTransformer.py", line 308, in __init__
    modules, self.module_kwargs = self._load_sbert_model(
                                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\SentenceTransformer.py", line 1728, in _load_sbert_model
    module = module_class(model_name_or_path, cache_dir=cache_folder, backend=self.backend, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\code\sentence-transformers\sentence_transformers\models\Transformer.py", line 81, in __init__
    self._load_model(model_name_or_path, config, cache_dir, backend, is_peft_model, **model_args)
  File "c:\code\sentence-transformers\sentence_transformers\models\Transformer.py", line 180, in _load_model
    self.auto_model = AutoModel.from_pretrained(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\models\auto\auto_factory.py", line 558, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.cache\huggingface\modules\transformers_modules\nomic-ai\nomic-bert-2048\eb02ceb48c1fdcc477ff1925c9732c379f0f0d1f\modeling_hf_nomic_bert.py", line 441, in from_pretrained   
    state_dict = state_dict_from_pretrained(model_name, dtype=dtype)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.cache\huggingface\modules\transformers_modules\nomic-ai\nomic-bert-2048\eb02ceb48c1fdcc477ff1925c9732c379f0f0d1f\modeling_hf_nomic_bert.py", line 84, in state_dict_from_pretrained
    resolved_archive_file = cached_file(model_name, weight_name, _raise_exceptions_for_missing_entries=False)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\tom\.conda\envs\sentence-transformers\Lib\site-packages\transformers\utils\hub.py", line 421, in cached_file
    raise EnvironmentError(
OSError: tomaarsen/nomic-embed-text-v1.5-copy is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'
If this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`

Here, the issue is that Nomic's custom code calls cached_file to load weights, but it doesn't (re-)use the token that was provided. So: it crashes.
However, in your case, I can't be sure whether the issue originates on the Sentence Transformers side, or in something from the remote repository. Are you willing to share more about your traceback and/or can you check whether the "faulty download call" originates in sentence-transformers or in one of the custom modeling files?

An interesting solution is to set the HF_TOKEN environment variable before loading the model, then whatever remote code can also access the token for loading extra things from the repository if it has to.

• Tom Aarsen

[deklanw]

This model is a simple fine-tune of https://huggingface.co/jinaai/jina-embeddings-v2-small-en. The need for trust_remote_code is inherited from it (in fact, I'd like to know how to remove this annoyance anyway!). I imagine you can study the issue with this model, making a private version if necessary.

EDIT: Ah, I see you've answered how to remove the necessity for trust_remote_code elsewhere . I'll probably do this for my own convenience, but this issue stands I guess

[tomaarsen]

Ah, yes. Thanks for the details.
The issue originates here: https://huggingface.co/jinaai/jina-bert-implementation/blob/main/modeling_bert.py#L1110
Just like with my nomic example, this custom code isn't expecting a private model/tokenizer to be loaded.

A couple of possible fixes:

1. Set HF_TOKEN, e.g. via huggingface-cli login if you want to set it "permanently", or you can set it temporarily for your script
2. Download the model locally and use it like described here: https://huggingface.co/jinaai/jina-embeddings-v2-small-en/discussions/19#671fa5e93fb4d4644abe37b4

cc @bwanglzu - this isn't actionable as it's probably tricky to fix, but it might be useful to know in case any of your users/clients encounter this themselves.
cc @zanussbaum idem with Nomic

• Tom Aarsen

[deklanw]

Thanks, I'll do one of these solutions. I have a remaining question but it's more appropriate for the jina discussion board.