Improved robustness of deep learning models through posterior agreement based model selection

Made with the lightning-hydra-template

Before you start

Before running the scripts you need to create a data, logs and outputs folder.

• Create a .env file to define the environment variables. Set up the following three directory paths
  • DATA_DIR: where the datasets and trained models will be stored
  • LOGS_DIR: where the logs of the training will be stored
  • OUTPUTS_DIR: where the command line outputs will be stored
• Create the three data folders according to the variables you have set up, or run the ./scripts/create_soft_links.sh to directly create soft links to the specified directory paths (useful if you want to store the data in remote locations)

Adversarial Learning

To create the adversarial datasets you can run

python src/generate_adv_data.py experiment=adv/generate_adv_data <option>=<value> ...

To replicate the adversarial experiments you can run

python src/train_pa.py experiment=adv/optimize_beta <option>=<value> ...

The additional parameters for both scripts are:

• model/adv/[email protected]: the attacked model (weak, or roubst)
• data/adv/[email protected]: the attack (PGD, FMN)
• data.attack.steps: the attack number of steps (tested with 1000)
• data.attack.batch_size: the attack batch size (tested with 1000)
• data.attack.epsilons: the attack power (only for PGD, tested with 0.0314, 0.0627 and 0.1255)
• data.adversarial_ratio: the attack adversarial ratio, in $[0, 1]$
• trainer: (optional) set it to cpu in order to disable GPU usage (for debugging)
• logger: (optional) set it to None to disable W&B logging

Domain Generalization

To replicate the adversarial experiments you can run

python src/train_pa.py experiment=adv/optimize_beta <option>=<val> ...

The additional parameters are:

• data.dg.ds1_env: the first dataset to compare with. Set it to test0
• data.dg.ds2_env: the other environments to compare with (test0, test1, test2, test3, test4, test5)
• data.dg.shift_ratio: the shift ratio, in $[0, 1]$
• model.dg.classifier.exp_name: the DG model (diagvib_weak, diagvib_robust)

Debugging parameters

• trainer: (optional) set it to cpu in order to disable GPU usage
• logger: (optional) set it to None to disable the default (W&B) logging

Multiple experiments

You can run multiple experiments by defining more values for one parameter, separated by a comma (e.g., data.attack.epsilons=0.0314,0.0627,0.1255) and by adding the option --multirun