Automatically tuning hyperparameters in Raster Vision

[natedailey7]

Hello,

I'm wondering if anyone has a recommendation for automatically tuning hyperparameters in Raster Vision (ex. learning rate, batch size, class loss weights). That is, so that different values are tried in order to find the best one. I've looked into Ray Tune, but I'm not sure how to implement something like this without getting deep into Raster Vision's guts (such as, in the train() function). Would anyone recommended any methods?

[AashishKumbashi]

I have a similar usecase and I ended up just adding a return statement in the train() function for the metrics and then used Optuna. to optimize based on the returned metric.

[natedailey7]

Thanks for your response! Would you mind sharing how you sent the training data from your train function into Optuna?

[AashishKumbashi]

Sure. In the train.py file, I added a line at the end to return metrics.

def train(self, epochs: Optional[int] = None):

        """Training loop that will attempt to resume training if appropriate."""
        start_epoch = self.get_start_epoch()

        if epochs is None:
            end_epoch = self.cfg.solver.num_epochs
        else:
            end_epoch = start_epoch + epochs

        if (start_epoch > 0 and start_epoch < end_epoch):
            log.info(f'Resuming training from epoch {start_epoch}')

        self.on_train_start()
        for epoch in range(start_epoch, end_epoch):
            log.info(f'epoch: {epoch}')
            train_metrics = self.train_epoch(
                optimizer=self.opt, step_scheduler=self.step_scheduler)
            if self.epoch_scheduler:
                self.epoch_scheduler.step()
            valid_metrics = self.validate_epoch(self.valid_dl)
            metrics = dict(epoch=epoch, **train_metrics, **valid_metrics)
            log.info(f'metrics:\n{pformat(metrics)}')

            self.on_epoch_end(epoch, metrics)
        return metrics

This is then called inside the objective function from Optuna.

def objective(trial):
    ###YOUR CODE HERE###
    map50 = (learner.train(epochs=epochs))['map50']
    return av_prec