Training Error #645

ANTZ314 · 2022-10-19T17:20:13Z

ANTZ314
Oct 19, 2022

Describe the bug

Runtime error at training ""

To Reproduce

Steps to reproduce the behavior:

No changes were made to the provided Colab workbook:

by @bth5

Expected behaviour

I ran the exact same sequence of commands on Google Colab and was able to compile multiple models (padim & cflow) and achieve various inferences from the trained model.
Command run:

!python tools/train.py --config anomalib/models/padim/config.yaml

ERROR response when running training command:

Traceback (most recent call last):
File "tools/train.py", line 71, in
train()
File "tools/train.py", line 49, in train
config = get_configurable_parameters(model_name=args.model, config_path=args.config)
File "/usr/local/lib/python3.7/dist-packages/anomalib/config/config.py", line 160, in get_configurable_parameters
config.metrics.threshold.pixel_default = config.metrics.threshold.image_default
File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 356, in getattr
key=key, value=None, cause=e, type_override=ConfigAttributeError
File "/usr/local/lib/python3.7/dist-packages/omegaconf/base.py", line 237, in _format_and_raise
type_override=type_override,
File "/usr/local/lib/python3.7/dist-packages/omegaconf/_utils.py", line 900, in format_and_raise
_raise(ex, cause)
File "/usr/local/lib/python3.7/dist-packages/omegaconf/_utils.py", line 798, in _raise
raise ex.with_traceback(sys.exc_info()[2]) # set env var OC_CAUSE=1 for full trace
File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 352, in getattr
key=key, default_value=DEFAULT_MARKER, validate_key=False
File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 443, in _get_impl
key=key, throw_on_missing_key=True, validate_key=validate_key
File "/usr/local/lib/python3.7/dist-packages/omegaconf/basecontainer.py", line 78, in _get_child
throw_on_missing_key=throw_on_missing_key,
File "/usr/local/lib/python3.7/dist-packages/omegaconf/dictconfig.py", line 480, in _get_node
raise ConfigKeyError(f"Missing key {key!s}")
omegaconf.errors.ConfigAttributeError: Missing key image_default
full_key: metrics.threshold.image_default
object_type=dict

Hardware and Software Configuration

GOOGLE COLAB
Python 3 Google Compute Engine backend (GPU)
[standard Colab account specs as of 2022-10-19]

Additional context

The suggested fix is to install an older version "omegaconf==2.0.0", however, this leads to a new error.

Answered by samet-akcay

Oct 21, 2022

@ANTZ314, @JACKYNIKK, the problem is that the jupyter notebook installs anomalib via pip, which installs v0.3.6 now. You clone the repo via !git clone https://github.com/openvinotoolkit/anomalib.git, which clones the latest commit, and causes conflict with the pypi version.

Solution
For now, if you clone and install anomalib as the following, the problem would temporarily be resolved.

Option - 1
Train the model via

!python tools/train.py --model padim

Option - 2
Install the library locally

!git clone https://github.com/openvinotoolkit/anomalib.git
%cd anomalib
!pip install -e .

We'll soon update the pypi version with v0.3.7, which would remedy this issue completely.

View full answer

samet-akcay · 2022-10-20T05:06:38Z

samet-akcay
Oct 20, 2022
Maintainer

@ANTZ314, this is probably due to one of our recent PRs #637. I'll check the notebooks and fix them if needed.

0 replies

JACKYNIKK · 2022-10-21T08:06:02Z

JACKYNIKK
Oct 21, 2022

Same happens for me

0 replies

samet-akcay · 2022-10-21T08:54:27Z

samet-akcay
Oct 21, 2022
Maintainer

interesting, we tried to ensure the backward compatibility, but apparently, it is broken. @JACKYNIKK, can you confirm you have this issue in the jupyter notebooks as well?

0 replies

JACKYNIKK · 2022-10-21T08:58:34Z

JACKYNIKK
Oct 21, 2022

I'm sorry, you mean if the same problem happens if i run the lines in my local jupyter notebook instead of running it online on colab? Or is something else meant?

0 replies

samet-akcay · 2022-10-21T09:00:04Z

samet-akcay
Oct 21, 2022
Maintainer

I meant Jupyter notebook in general, both local or cloud? Or do you also have this issue when you run train.py as well?

0 replies

JACKYNIKK · 2022-10-21T09:16:02Z

JACKYNIKK
Oct 21, 2022

Im not sure if this is what was meant.
Anyway i tried to create a new enviroment on anaconda on my pc. Then openend a jupyter notebook in this enviroment and cloned the repository with: "!git clone https://github.com/openvinotoolkit/anomalib.git". After that i ran:
"!pip install anomalib" and got into the right folder with :"cd anomalib".
If i run: "!python tools/train.py --config anomalib/models/padim/config.yaml" at this point as in the colab i get the same exact error as on colab. i report a screen here:

Even if i use this repository i'm unexperienced with python, colab, cloud and so on and i'm not sure i understood what was meant. Hope this was it, otherwise let me know if i should try something else.

0 replies

samet-akcay · 2022-10-21T10:16:45Z

samet-akcay
Oct 21, 2022
Maintainer

@ANTZ314, @JACKYNIKK, the problem is that the jupyter notebook installs anomalib via pip, which installs v0.3.6 now. You clone the repo via !git clone https://github.com/openvinotoolkit/anomalib.git, which clones the latest commit, and causes conflict with the pypi version.

Solution
For now, if you clone and install anomalib as the following, the problem would temporarily be resolved.

Option - 1
Train the model via

!python tools/train.py --model padim

Option - 2
Install the library locally

!git clone https://github.com/openvinotoolkit/anomalib.git
%cd anomalib
!pip install -e .

We'll soon update the pypi version with v0.3.7, which would remedy this issue completely.

5 replies

ANTZ314 Oct 26, 2022
Author

@samet-akcay Sorry, I finally had a chance to attempt Option-1 & Option-2, and it still looks like the same Error as above with both training commands.

What I haven't mentioned previously is that there are 2 errors at the end of each install command. For the sake of space, I have cut out all of the "successful messages" with just the command and error/warning as follows:

Command 1
!git clone https://github.com/openvinotoolkit/anomalib.git %cd anomalib !pip install -e .

ERROR
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchaudio 0.12.1+cu113 requires torch==1.12.1, but you have torch 1.12.0 which is incompatible. Successfully installed - multiple packages here.... WARNING: The following packages were previously imported in this runtime: [matplotlib,mpl_toolkits,pydevd_plugins] You must restart the runtime in order to use newly installed versions.

Command 2
!pip install anomalib

ERROR
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchaudio 0.12.1+cu113 requires torch==1.12.1, but you have torch 1.11.0 which is incompatible. Successfully installed anomalib-0.3.6 torch-1.11.0 torchtext-0.12.0 torchvision-0.12.0

ANTZ314 Oct 26, 2022
Author

Is there a way to directly download / '!git clone' the older version that was compatible with the current pypi version?

ANTZ314 Oct 26, 2022
Author

Okay, one more thing worth noting is that after cloning the project, you "pip upgrade" the torch packages, but right after the "pip install" of anomalib, all of the torch packages have reverted back to their previous versions:

After "!git clone..." toch versions:

pytorch-lightning==1.6.5
torch==1.11.0
torchaudio @ https://download.pytorch.org/whl/cu113/torchaudio-0.12.1%2Bcu113-cp37-cp37m-linux_x86_64.whl
torchmetrics==0.9.3
torchsummary==1.5.1
torchtext==0.12.0
torchvision==0.12.0

After upgrading all torch packages:

pytorch-lightning==1.6.5
torch==1.12.1
torchaudio @ https://download.pytorch.org/whl/cu113/torchaudio-0.12.1%2Bcu113-cp37-cp37m-linux_x86_64.whl
torchmetrics==0.9.3
torchsummary==1.5.1
torchtext==0.13.1
torchvision==0.13.1

After "!pip install anomalib":

pytorch-lightning==1.6.5
torch==1.11.0
torchaudio @ https://download.pytorch.org/whl/cu113/torchaudio-0.12.1%2Bcu113-cp37-cp37m-linux_x86_64.whl
torchmetrics==0.9.3
torchsummary==1.5.1
torchtext==0.12.0
torchvision==0.12.0

samet-akcay Oct 28, 2022
Maintainer

It's probably because when you tried pip install anomalib, it was trying to install v0.3.6, whose requirements are as follows:

anomalib/requirements/base.txt

Lines 13 to 15 in bd03b97

    
           torchmetrics>=0.9.1,<=0.9.3 
        
           torchvision>=0.9.1,<=0.12.0 
        
           torchtext>=0.9.1,<=0.12.0

We have just released v0.3.7, which should address theses issues since this release addresses those compatibility issues you have had. Can you try again please?

ANTZ314 Oct 29, 2022
Author

@samet-akcay YEEESSS!!!
It's back to training again! Looks like v0.3.7 has sorted out the issue.
Thank you for your assistance and hard work.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Training Error #645

{{title}}

Replies: 7 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Training Error #645

ANTZ314 Oct 19, 2022

Replies: 7 comments · 5 replies

samet-akcay Oct 20, 2022 Maintainer

JACKYNIKK Oct 21, 2022

samet-akcay Oct 21, 2022 Maintainer

JACKYNIKK Oct 21, 2022

samet-akcay Oct 21, 2022 Maintainer

JACKYNIKK Oct 21, 2022

samet-akcay Oct 21, 2022 Maintainer

ANTZ314 Oct 26, 2022 Author

ANTZ314 Oct 26, 2022 Author

ANTZ314 Oct 26, 2022 Author

samet-akcay Oct 28, 2022 Maintainer

ANTZ314 Oct 29, 2022 Author

ANTZ314
Oct 19, 2022

Replies: 7 comments 5 replies

samet-akcay
Oct 20, 2022
Maintainer

JACKYNIKK
Oct 21, 2022

samet-akcay
Oct 21, 2022
Maintainer

JACKYNIKK
Oct 21, 2022

samet-akcay
Oct 21, 2022
Maintainer

JACKYNIKK
Oct 21, 2022

samet-akcay
Oct 21, 2022
Maintainer

ANTZ314 Oct 26, 2022
Author

ANTZ314 Oct 26, 2022
Author

ANTZ314 Oct 26, 2022
Author

samet-akcay Oct 28, 2022
Maintainer

ANTZ314 Oct 29, 2022
Author