Problem with low performance on custom dataset and HPO

[JACKYNIKK]

I don't know if it is appropriate to ask questions on how to use the repo in this issues section(i'm new to github). If not please let me know where i can find support on this.

Back to the issue:
I followed the tutorial to use the repo on my custom dataset:
https://github.com/openvinotoolkit/anomalib/tree/development/docs/blog/001-train-custom-dataset

The dataset i use is structured like this:

with GOOD_PULITO containing 252 images of normal type(224x224 pixels in jpg format) and BAD_PULITO containing 228 images of anomalous type(same size and format).

I tested training and inference on all models and everything works fine but the results are not that good(AUROC of about 0.8 or less).
So i thougth i should try HPO as suggested in the tutorial.

The command i use to do the HPO is (for example on dfkde):
python tools/hpo/sweep.py --model dfkde --model_config anomalib/models/dfkde/config_custom.yaml -
-sweep_config tools/hpo/sweep.yaml

it works fine and i'm able to visualize the results on wandb but as said these results are poor on my dataset.

So 2 questions arise from my side:

first one:
is the default sweep.yaml file here(which i suppose is written for padim):
sweep.txt
meant to be used on all models without changes? And how would i need to change it to use it for example on GANomaly or dfkde?

Second question:
I was wondering how to change the sweep.yaml file to increase the perfomances of the HPO (in other words: which other Hyperparameters could i include in the sweep.yaml file other than the backbone and image_size?)

Thank you in advance

[djdameln]

I don't know if it is appropriate to ask questions on how to use the repo in this issues section(i'm new to github). If not please let me know where i can find support on this.

Of course we are always happy to answer questions. The discussions section (especially the Q&A category) is a bit more suitable for these types of questions than the issues section, so I converted your issue into a discussion.

To answer your question I'm tagging @ashwinvaidya17, who implemented the HPO functionality and is a bit more knowledgeable on this topic.

[JACKYNIKK]

Thank you for moving the question in the most appropiate place.

For example i saw on the wandb documentation that momentum and learning rate are accepted hyperparameters but still it is not clear to me if there's a complete list of hyperparameters i can add to the sweep.yaml file to try and get better performance.
Anyway thank you for now

[ashwinvaidya17]

Hi @JACKYNIKK you can have a look at the parameters in the model and dataset sections of the respective configs. Here is what's there in the ganomaly's config file (https://github.dev/openvinotoolkit/anomalib/blob/3ae63a1135f33972d3b7a0aa1d17d3c907fce3ce/anomalib/models/ganomaly/config.yaml)

model:
  name: ganomaly
  latent_vec_size: 100
  n_features: 64
  extra_layers: 0
  add_final_conv: true
  early_stopping:
    patience: 3
    metric: image_AUROC
    mode: max
  lr: 0.0002
  beta1: 0.5
  beta2: 0.999
  wadv: 1
  wcon: 50
  wenc: 1
  normalization_method: none

You can use HPO on all the parameters (except name) mentioned in this section (and the ones in dataset). For example you can try

model:
    latent_vec_size:
        values: [50, 100, 200]
    n_features:
        values: [32, 64, 128]
   lr:
        min: 0.0001
        max: 0.1
...

Let me know if this example is clear. You can also refer to the wandb tutorial to get some idea on how to configure the parameters. https://docs.wandb.ai/guides/sweeps/quickstart

[JACKYNIKK]

Sorry for the late reply. Thank you so much. The example is more than clear and the HPO works fine now also on GANomaly. Great.