FutureWarning: You are using torch.load with weights_only=False

[mskaif]

Describe the bug
FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
state = torch.load(filename, lambda storage, loc: storage)

This warning is triggered by all torch.load used in stanza. The issue does not cause any problem with data processing at the moment but the long warnings are distracting.

To Reproduce
Steps to reproduce the behavior:

1. upgrade torch to 2.4.1

Expected behavior
no error

Environment (please complete the following information):

• OS: Windows
• Python version: python 3.12.7
• Stanza version: 1.9.2

[mskaif]

The error can be suppressed by using the following before calling stanza functions but is not a solution

import warnings
warnings.simplefilter(action='ignore', category=FutureWarning)

source: ultralytics/ultralytics#14994 (comment)

[AngledLuffa]

Aware of it. There's a limitation where we are saving plenty of things other than weights in the current file. Config strings and numbers, mostly. Would those still work?

…

On Tue, Oct 22, 2024, 10:19 PM mskaif ***@***.***> wrote: The error can be suppressed by using the following before calling stanza functions but is not a solution import warnings warnings.simplefilter(action='ignore', category=FutureWarning) source: ultralytics/ultralytics#14994 (comment) <ultralytics/ultralytics#14994 (comment)> — Reply to this email directly, view it on GitHub <#1429 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2AYWPIB3WEPPBSGN337ALZ44WXBAVCNFSM6AAAAABQN6FUXGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZQHEZTAMBQGY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[AngledLuffa]

Some of the models can be updated to use weights_only=True right away, but others require resaving with enums or other data structures removed. Will have to investigate some more.

[mskaif]

Some of the models can be updated to use weights_only=True right away, but others require resaving with enums or other data structures removed. Will have to investigate some more.

sorry for not getting back earlier. I'm using the built-in models like so:
STANZA_PIPE = stanza.Pipeline(
lang="en",
dir=settings.STANZA_DATA_DIR,
processors="tokenize,mwt,pos",
download_method=None,
use_gpu=False,
)

affected from the pipeline are:
tokenization\trainer.py:82
mwt\trainer.py:201
pos\trainer.py:139
common\pretrain.py:56
common\char_model.py:271

Thank you for the commit!