GPU optimization and changing batch size with AutoSeg3D

[dmolony3]

Hi,

I'm having difficulty understanding and 1. changing batch size and 2. using GPU optimization with AutoSeg3D. My problem is that I don't seem to be able to change batch size (either manually or through GPU optimization) and I'm not sure what output file confirms the batch size used in the simulation.

Specifically, I'm wondering

• What is the difference between the hyperparameters batch_size and num_images_per_batch?
• How can I see the actual batch size used in the training? I have tried setting this in the input.yaml as well as passing it via the train_param to algo.train(). My GPU RAM usage does not change.
• My understanding is that I can use GPU optimization to find the best batch size that fits the GPU. Does the GPU optimization produce an output file detailing the process, where can I find this?

Below is my code

# prepare input yaml configuration
dataroot = "/kaggle/temp"
datalist_file = "LAA_folds.json"

task_name= "LAA"
input_cfg = {
    "name": task_name,  # optional, it is only for your own record
    "task": "segmentation",  # optional, it is only for your own record
    "modality": "CT",  # required
    "datalist": datalist_file,  # required
    "dataroot": dataroot,  # required
    "batch_size": 4
}
input_file = "./input.yaml"
ConfigParser.export_config_file(input_cfg, input_file)

work_dir = "/kaggle/working/output_folder"

if not os.path.isdir(work_dir):
    os.makedirs(work_dir)
datastats_file = os.path.join(work_dir, "data_stats.yaml")
analyser = DataAnalyzer(datalist_file, dataroot, output_path=datastats_file)
datastat = analyser.get_all_case_stats()


bundle_generator = BundleGen(
    algos=['segresnet'],
    algo_path=work_dir,
    data_stats_filename=datastats_file,
    data_src_cfg_name=input_file,
)

gpu_customization_specs = {
    "universal": {"num_trials": 20, "range_num_images_per_batch": [2, 8], "range_num_sw_batch_size": [1, 40]}
}
bundle_generator.generate(
    work_dir,
    num_fold=5,
    gpu_customization=True,
    gpu_customization_specs=gpu_customization_specs,
)

history = bundle_generator.get_history()
export_bundle_algo_history(history)  # save the Algo objects

max_epochs = 10  

train_param = {
        "num_epochs": max_epochs,
        "batch_size": 4
}

history = import_bundle_algo_history(work_dir, only_trained=False)
for algo_dict in history:
    algo = algo_dict[AlgoKeys.ALGO]
    algo.train(train_param)  # can use default params by `algo.train()`
    acc = algo.get_score()
    algo_to_pickle(algo, template_path=algo.template_path, best_metric=acc)

[KumoLiu]

Hi @dmolony3, please use num_images_per_batch to modify the batch size.

[KumoLiu]

And yes it can automatically generate several parameters, you can get the real hyper-parameters in each algo folder.
And if you want modify the batch size, you may need to set auto_scale_allowed to False.
You can refer this tutorial for more details.

https://github.com/Project-MONAI/tutorials/blob/main/auto3dseg/notebooks/auto_runner.ipynb

Thanks.