Model Selection APIs

Overview

The user uses a set of high-level model selection APIs to specify the AutoML procedure and the workload. The system then automatically takes care of scheduling and execution via MOP. The implementation of this layer is located at cerebro.tune.

Implementation of Model Selection APIs

The most important API components are the ModelSelection classes; each of them implements an AutoML procedure/model selection heuristic. They take in the following user inputs:

• Execution backend
• Storage
• Parameter search grid
• Other procedure-specific parameters

They implement the following functionalities:

• Creation of the model selection workload from the parameter grid
• Epoch-level management of execution
• Management of the workload, if the AutoML procedure requires dynamic updates to the workload
• Logging the performance metrics

Extension

To implement other AutoML procedures other than the ones defined in cerebro.tune, extend the base class cerebro.tune.base.ModelSelection and override the following virtual functions:

• __init__: initializer function that takes in user inputs and generates the (initial) model selection workload, and also handles initialization of the Cerebro driver and workers
• _fit_on_prepared_data: the function that invokes the Backend for training. It manages the epoch-level training loop and, if necessary, updates the model selection workload at each epoch-end. Finally, it handles the logging of the performance metrics and returns the model selection results