Define custom scorer outside CustomXGBoostRegressor class (default) to facilitate additional scorer options

[khsjursen]

The scorer currently used with CustomXGBoostRegressor in GridSearchCV (to evaluate validation performance for each fold) is implemented inside the class such that it is used by default when passing scoring = None to GridSearchCV. The scorer is currently a custom MSE scorer that aggregates according to unique ids.

It should be possible to define custom scoring functions outside of the CustomXGBoostRegressor class such that the user can choose which scorer to use in GridSearchCV, e.g. scoring = custom_mse. Using default scorers from sklearn does not work because the scorer needs to aggregate y_pred from the trained model/fold and get the mean of y_val for each unique id.

See: https://scikit-learn.org/stable/modules/model_evaluation.html#implementing-your-own-scoring-object

make_scorer from Scikit-learn should work, but this is NOT working:

from sklearn.metrics import make_scorer

def custom_mse_scorer(y_true, y_pred, X):
    # Extract metadata
    metadata = X[:, -3:] 

    all_pred_agg = []
    all_true_mean = []
    
    unique_ids = np.unique(metadata[:, 0]) # ID is first column of metadata
    
    # Loop over each unique ID to aggregate/get mean
    for uid in unique_ids:
        indexes = metadata[:, 0] == uid
        
        # Aggregate predictions for the current ID
        y_pred_agg = np.sum(y_pred[indexes])
        
        # Get mean of true values for the current ID
        y_true_mean = np.mean(y_true[indexes])

        all_pred_agg.append(y_pred_agg)
        all_true_mean.append(y_true_mean)
        
        #mse += (y_pred_agg - y_true_mean) ** 2

    all_pred_agg = np.array(all_pred_agg)
    all_true_mean = np.array(all_true_mean)

    # Compute mse 
    mse = ((all_pred_agg - all_true_mean) ** 2).mean()
    #mse /= len(unique_ids)    
    
    return -mse  # Return negative because GridSearchCV maximizes score

# Need to use the make_scorer function to turn the custom evaluation function into a scorer object
# Wrap custom mse scorer to pass metadata 
def make_custom_mse_scorer():
    """Generates a customized scorer callable as required by GridSearchCV"""
    return make_scorer(custom_mse_scorer, greater_is_better=False, needs_proba=False, needs_X=True)

custom_mse_scorer = make_custom_mse_scorer()