TypeError: expected np.ndarray (got Tensor)

[topl0305]

Describe the bug
Was trying to use pretrained model https://huggingface.co/stanfordnlp/stanza-lt
With a lot of issues, like stanza.download("lt") constantly crashing, I was forced to do it manually. So, installed and downloaded everything and used next piece of code to get the bug

import stanza
config = {
'processors': 'tokenize,pos',
'lang': 'lt',
'tokenize_model_path': './stanza_resources/lt/tokenize/alksnis.pt',
'pos_model_path': './stanza_resources/lt/pos/alksnis_nocharlm.pt',
'pos_pretrain_path': './stanza_resources/lt/pretrain/fasttextwiki.pt',
'tokenize_pretokenized': True,
'download_method': None
}

nlp = stanza.Pipeline(**config) # initialize neural pipeline
doc = nlp("Kur einam mes su Knysliuku, didžiulė paslaptis") # run annotation over a sentence
print(doc)


Expected behavior
The result shoud be obvious:

[
[
{
"id": 1,
"text": "Kur",
"upos": "ADV",
"xpos": "prm.l.lrgin.",
"feats": "Degree=Pos|PronType=Int,Rel",
"misc": "",
"start_char": 0,
"end_char": 3
},
...
]

Environment (please complete the following information):

• OS: Windows 10
• Python 3.10.5
• stanza 1.9.2
• numpy 2.1.2

Additional context
At least it works after patching code in file stanza/models/pos/model.py
~90 line self.add_unsaved_module('pretrained_emb', nn.Embedding.from_pretrained(torch.from_numpy(emb_matrix), freeze=True))
to

if type(emb_matrix) == torch.Tensor:
self.add_unsaved_module('pretrained_emb', nn.Embedding.from_pretrained(emb_matrix, freeze=True))
else:
self.add_unsaved_module('pretrained_emb', nn.Embedding.from_pretrained(torch.from_numpy(emb_matrix), freeze=True))

Not sure who is culprit - library or model.

[AngledLuffa]

ultimately the problem here is we modified the models for the upcoming version 1.10, and you're downloading the new models with the old code. you could use the dev branch or download the version 1.9 models directly from HF if you're sure you need to do it manually

With a lot of issues, like stanza.download("lt") constantly crashing, I was forced to do it manually.

"crashing" how? like with a bad connection? it doesn't "crash" when i run it

you also don't need to do any of that

just run

nlp = stanza.Pipeline("lt", processors="tokenize,pos", tokenize_pretokenized=True)

it should automatically download just the models you need for the right version

[topl0305]

If I'm using direct download - stanza.download('lt')
I get next error

Traceback (most recent call last):
  File "C:/Users/***/Desktop/test_nlp.py", line 2, in <module>
    stanza.download('lt') # download English model
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\resources\common.py", line 599, in download
    request_file(
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\resources\common.py", line 159, in request_file
    assert_file_exists(path, md5, alternate_md5)
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\resources\common.py", line 112, in assert_file_exists
    raise ValueError("md5 for %s is %s, expected %s" % (path, file_md5, md5))
ValueError: md5 for C:\Users\***\stanza_resources\lt\default.zip is 36e9cd4989fac42001d585dc514c2020, expected 3b1725c28eeed0cdf734bd92ec82f927

This is log file:
log.txt

Was testing your suggestion -- nlp = stanza.Pipeline("lt", processors="tokenize,pos", tokenize_pretokenized=True)

Traceback (most recent call last):
  File "C:/Users/***/Desktop/test_nlp.py", line 7, in <module>
    nlp = stanza.Pipeline("lt", processors="tokenize,pos", tokenize_pretokenized=True)
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\pipeline\core.py", line 252, in __init__
    download_models(download_list,
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\resources\common.py", line 540, in download_models
    request_file(
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\resources\common.py", line 159, in request_file
    assert_file_exists(path, md5, alternate_md5)
  File "C:\Users\***\AppData\Local\Programs\Python\Python310\lib\site-packages\stanza\resources\common.py", line 112, in assert_file_exists
    raise ValueError("md5 for %s is %s, expected %s" % (path, file_md5, md5))
ValueError: md5 for C:\Users\***\stanza_resources\lt\pretrain\fasttextwiki.pt is 6996c18339716076308d957354340a61, expected 89420a04d9c0b31feb5598e17eb52f8f

[AngledLuffa]

That's pretty weird. If I use the github repo main branch (which is 1.9.2), download successfully downloads a file with the following md5sum, which is the expected value:

[john@localhost stanza]$ md5sum /home/john/stanza_resources/lt/default.zip
3b1725c28eeed0cdf734bd92ec82f927  /home/john/stanza_resources/lt/default.zip

I can switch branches back & forth between main & dev, and it overwrites the old models when trying to download again. At no point does it download a model with md5sum 36e9cd4989fac42001d585dc514c2020 This works on both Linux and Windows

Is it possible the download was interrupted and it got a corrupted file?

At any rate, I suggest deleting those incorrect files and trying again.

[AngledLuffa]

There is a new version pushed, with models that should be updated with a new internal format.

[topl0305]

Don't mind the last message (deleted it). Was trying another library and it pushed torch 2.0.1. Now everything works fine.

[AngledLuffa]

phew, already had to make one hotfix for SNAFU

[AngledLuffa]

everything = original problem is also resolved with the new version, hopefully?

[topl0305]

Suggestion. When loading stanza, check for dependencies versions and show a warning/error. Sometimes you can just overlook changes with all dependencies.

And yes, no more warnings or errors.

[AngledLuffa]

I think the hope is the dependency manager will make those warnings or errors when upgrading :) for example, i did get exactly such a warning when i tried to update numpy without also updating scipy

[topl0305]

Hopefully, this will be the last message from me. Checked METADATA file in stanza dist:

Metadata-Version: 2.1
Name: stanza
Version: 1.10.1
.....
Requires-Dist: numpy
Requires-Dist: torch>=1.3.0
...
But from my experience not any numpy version works (i'm using 2.2.0) and, as you already know, torch should be > 2.0.1. I'm currently using 2.5.1. So, it would be meaningfull to check the list of requirements and renew it.

[AngledLuffa]

Good point. The weights_only parameter was actually added in torch 1.13, and I don't know of any other features we've used since then:

https://pytorch.org/docs/1.13/generated/torch.load.html

Is there a reason to specify 2.0.1 that you know of?

[AngledLuffa]

to be entirely honest i got the chance to share your pain - tried updating to numpy 2 on my desktop, found that meant i needed to update to a later torch, found that meant i needed to update cuda & cudnn and then install libnccl, which had never been a thing before. so my efforts to track down your issue cost me a couple hours of fiddling with all these things

[topl0305]

Yes, you nailed it :) I had numpy 2.2.0 + torch 2.0.1 for setup and when tried to run stanza i got error about not having numpy or something. If i remember correctly it was error related to torch.numpy(), but not sure atm.