You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Jul 29, 2024. It is now read-only.
Should stratified cross-validation based on the Treatment vs Outcome 2x2 matrix split be used when performing a grid search to ensure that each fold follows the same distribution used in the overall data? If this is not the case, and cross validation is used for hyperparameter search, should we expect that the scores used for evaluating each fold, using qini coefficient as an example, indeed represent the qini coefficient for the overall training dataset? Putting it differently, an increase in number of folds may affect the stability of the uplift score in each fold.
Another closely related question, given the following two cross validation outputs, which one should we prefer? Higher mean score across folds, or uniform score across folds?
Number of cross folds: 4
Split_1_Score | split_2_Score | Split_3_Score | Split_4_Score | Mean_Score
0.4 0.9 -0.2 -0.3 | 0.2
Number of cross folds: 2
Split_1_Score | split_2_Score | Mean_Score
0.12 0.16 | 0.14
The text was updated successfully, but these errors were encountered:
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Should stratified cross-validation based on the Treatment vs Outcome 2x2 matrix split be used when performing a grid search to ensure that each fold follows the same distribution used in the overall data? If this is not the case, and cross validation is used for hyperparameter search, should we expect that the scores used for evaluating each fold, using qini coefficient as an example, indeed represent the qini coefficient for the overall training dataset? Putting it differently, an increase in number of folds may affect the stability of the uplift score in each fold.
Another closely related question, given the following two cross validation outputs, which one should we prefer? Higher mean score across folds, or uniform score across folds?
Number of cross folds: 4
Split_1_Score | split_2_Score | Split_3_Score | Split_4_Score | Mean_Score
0.4 0.9 -0.2 -0.3 | 0.2
Number of cross folds: 2
Split_1_Score | split_2_Score | Mean_Score
0.12 0.16 | 0.14
The text was updated successfully, but these errors were encountered: