Skip to content

Commit

Permalink
nabenabe0928: Refactoring base dataset splitting functions (#106)
Browse files Browse the repository at this point in the history
Github Actions committed Apr 28, 2021
1 parent 4bfb39c commit 6d73691
Showing 22 changed files with 452 additions and 404 deletions.
Binary file modified refactor_development/_downloads/2f0f54a9831653ac5c12ee0e06324a62/basics_tabular_jupyter.zip
View file
Binary file not shown.
Binary file modified refactor_development/_downloads/8ef6602cf8ed40221edd89244fb031af/basics_tabular_python.zip
View file
Binary file not shown.
Binary file modified refactor_development/_downloads/b38295cf56f02b1c4547385a8f389f90/advanced_tabular_python.zip
View file
Binary file not shown.
Binary file modified ...ctor_development/_downloads/df038041811db9bd567de83692a8f994/advanced_tabular_jupyter.zip
View file
Binary file not shown.
Binary file modified refactor_development/_images/sphx_glr_example_custom_configuration_space_thumb.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified refactor_development/_images/sphx_glr_example_resampling_strategy_thumb.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified refactor_development/_images/sphx_glr_example_tabular_classification_thumb.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified refactor_development/_images/sphx_glr_example_tabular_regression_thumb.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
138 changes: 25 additions & 113 deletions ...evelopment/_sources/advanced_tabular/example_custom_configuration_space.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -46,7 +46,7 @@ the search. Currently, there are two changes that can be made to the space:-

.. code-block:: none
<smac.runhistory.runhistory.RunHistory object at 0x7f2ad8808a90> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7fd58f4d3d00> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -80,7 +80,7 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.002296924591064453, budget=0), TrajEntry(train_perf=0.14035087719298245, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001336812973022461, budget=0), TrajEntry(train_perf=0.14619883040935677, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -114,109 +114,19 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=6.5182719230651855, wallclock_time=8.156251907348633, budget=5.555555555555555), TrajEntry(train_perf=0.14035087719298245, incumbent_id=2, incumbent=Configuration:
data_loader:batch_size, Value: 475
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
imputer:categorical_strategy, Value: 'most_frequent'
imputer:numerical_strategy, Value: 'constant_zero'
lr_scheduler:__choice__, Value: 'NoScheduler'
network_backbone:MLPBackbone:activation, Value: 'tanh'
network_backbone:MLPBackbone:dropout_1, Value: 0.579891279191762
network_backbone:MLPBackbone:dropout_2, Value: 0.43202885747368863
network_backbone:MLPBackbone:dropout_3, Value: 0.2053050533304992
network_backbone:MLPBackbone:dropout_4, Value: 0.3628626567848122
network_backbone:MLPBackbone:dropout_5, Value: 0.000687232634536894
network_backbone:MLPBackbone:dropout_6, Value: 0.30779918180581656
network_backbone:MLPBackbone:dropout_7, Value: 0.4566654226669556
network_backbone:MLPBackbone:num_groups, Value: 7
network_backbone:MLPBackbone:num_units_1, Value: 749
network_backbone:MLPBackbone:num_units_2, Value: 751
network_backbone:MLPBackbone:num_units_3, Value: 759
network_backbone:MLPBackbone:num_units_4, Value: 664
network_backbone:MLPBackbone:num_units_5, Value: 219
network_backbone:MLPBackbone:num_units_6, Value: 757
network_backbone:MLPBackbone:num_units_7, Value: 1005
network_backbone:MLPBackbone:use_dropout, Value: True
network_backbone:__choice__, Value: 'MLPBackbone'
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.9640640623783606
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.017233504391813814
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.24122690885917664
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.31247176333246596
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.41504826813841933
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.8395119637200936
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.8208414027523236
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.4284420622613293
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 7
network_embedding:__choice__, Value: 'LearnedEntityEmbedding'
network_head:__choice__, Value: 'fully_connected'
network_head:fully_connected:num_layers, Value: 1
network_init:KaimingInit:bias_strategy, Value: 'Zero'
network_init:__choice__, Value: 'KaimingInit'
optimizer:AdamOptimizer:beta1, Value: 0.9770847327434384
optimizer:AdamOptimizer:beta2, Value: 0.9710627513919582
optimizer:AdamOptimizer:lr, Value: 0.00010844892447274338
optimizer:AdamOptimizer:weight_decay, Value: 0.05048412416506887
optimizer:__choice__, Value: 'AdamOptimizer'
scaler:Normalizer:norm, Value: 'max'
scaler:__choice__, Value: 'Normalizer'
trainer:StandardTrainer:weighted_loss, Value: False
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=11, ta_time_used=135.23650550842285, wallclock_time=163.7278196811676, budget=16.666666666666664), TrajEntry(train_perf=0.13450292397660824, incumbent_id=3, incumbent=Configuration:
data_loader:batch_size, Value: 174
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:PowerTransformer:standardize, Value: True
feature_preprocessor:__choice__, Value: 'PowerTransformer'
imputer:categorical_strategy, Value: 'most_frequent'
imputer:numerical_strategy, Value: 'most_frequent'
lr_scheduler:ExponentialLR:gamma, Value: 0.9015102188730918
lr_scheduler:__choice__, Value: 'ExponentialLR'
network_backbone:MLPBackbone:activation, Value: 'relu'
network_backbone:MLPBackbone:num_groups, Value: 2
network_backbone:MLPBackbone:num_units_1, Value: 738
network_backbone:MLPBackbone:num_units_2, Value: 371
network_backbone:MLPBackbone:use_dropout, Value: False
network_backbone:__choice__, Value: 'MLPBackbone'
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.5026052624432612
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.7197339745319954
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.9509463920492078
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.5338374300900856
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.2115815132516191
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.3101651859744802
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.724733217572461
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.1459031891772793
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 4
network_embedding:__choice__, Value: 'LearnedEntityEmbedding'
network_head:__choice__, Value: 'fully_connected'
network_head:fully_connected:activation, Value: 'tanh'
network_head:fully_connected:num_layers, Value: 3
network_head:fully_connected:units_layer_1, Value: 212
network_head:fully_connected:units_layer_2, Value: 414
network_init:NoInit:bias_strategy, Value: 'Normal'
network_init:__choice__, Value: 'NoInit'
optimizer:RMSpropOptimizer:alpha, Value: 0.5152986504870539
optimizer:RMSpropOptimizer:lr, Value: 0.003121222619830083
optimizer:RMSpropOptimizer:momentum, Value: 0.18725278090773345
optimizer:RMSpropOptimizer:weight_decay, Value: 0.0065085103879551805
optimizer:__choice__, Value: 'RMSpropOptimizer'
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: False
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=17, ta_time_used=213.5607807636261, wallclock_time=264.5632119178772, budget=50.0)]
{'accuracy': 0.8554913294797688}
, ta_runs=1, ta_time_used=4.047108173370361, wallclock_time=5.4710657596588135, budget=5.555555555555555)]
{'accuracy': 0.8901734104046243}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:----------------------------------------------------------|---------:|
| 0 | None | RFClassifier | 0.26 |
| 1 | None | ExtraTreesClassifier | 0.24 |
| 2 | SimpleImputer,OneHotEncoder,MinMaxScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 3 | None | CatBoostClassifier | 0.1 |
| 4 | None | KNNClassifier | 0.1 |
| 5 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 6 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 7 | SimpleImputer,OneHotEncoder,Normalizer,NoFeaturePreprocessing | embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 8 | None | LGBMClassifier | 0.02 |
| 9 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
<smac.runhistory.runhistory.RunHistory object at 0x7f2ad802a5e0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
| 0 | None | CatBoostClassifier | 0.28 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.2 |
| 2 | None | ExtraTreesClassifier | 0.18 |
| 3 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
| 4 | None | KNNClassifier | 0.06 |
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 6 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 7 | None | RFClassifier | 0.04 |
<smac.runhistory.runhistory.RunHistory object at 0x7fd57e5f4d00> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'NoEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -248,7 +158,7 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001631021499633789, budget=0), TrajEntry(train_perf=0.21052631578947367, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0012526512145996094, budget=0), TrajEntry(train_perf=0.19298245614035092, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: 'NoEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -280,15 +190,17 @@ the search. Currently, there are two changes that can be made to the space:-
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=5.196228265762329, wallclock_time=6.9113147258758545, budget=5.555555555555555)]
, ta_runs=1, ta_time_used=3.548128843307495, wallclock_time=4.968382835388184, budget=5.555555555555555)]
{'accuracy': 0.8728323699421965}
| | Preprocessing | Estimator | Weight |
|---:|:----------------|:---------------------|---------:|
| 0 | None | ExtraTreesClassifier | 0.34 |
| 1 | None | CatBoostClassifier | 0.28 |
| 2 | None | RFClassifier | 0.2 |
| 3 | None | KNNClassifier | 0.16 |
| 4 | None | SVC | 0.02 |
| | Preprocessing | Estimator | Weight |
|---:|:-------------------------------------------------|:-------------------------------------------------------------------|---------:|
| 0 | None | CatBoostClassifier | 0.24 |
| 1 | None | RFClassifier | 0.22 |
| 2 | None | ExtraTreesClassifier | 0.18 |
| 3 | None | KNNClassifier | 0.14 |
| 4 | None | LGBMClassifier | 0.1 |
| 5 | SimpleImputer,NoEncoder,MinMaxScaler,KitchenSink | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 6 | None | SVC | 0.06 |
@@ -417,7 +329,7 @@ the search. Currently, there are two changes that can be made to the space:-
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 11 minutes 46.927 seconds)
**Total running time of the script:** ( 11 minutes 24.080 seconds)


.. _sphx_glr_download_advanced_tabular_example_custom_configuration_space.py:
180 changes: 142 additions & 38 deletions refactor_development/_sources/advanced_tabular/example_resampling_strategy.rst.txt
View file

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions refactor_development/_sources/advanced_tabular/sg_execution_times.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -5,10 +5,10 @@

Computation times
=================
**22:27.526** total execution time for **advanced_tabular** files:
**21:13.468** total execution time for **advanced_tabular** files:

+--------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_advanced_tabular_example_custom_configuration_space.py` (``example_custom_configuration_space.py``) | 11:46.927 | 0.0 MB |
| :ref:`sphx_glr_advanced_tabular_example_custom_configuration_space.py` (``example_custom_configuration_space.py``) | 11:24.080 | 0.0 MB |
+--------------------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_advanced_tabular_example_resampling_strategy.py` (``example_resampling_strategy.py``) | 10:40.599 | 0.0 MB |
| :ref:`sphx_glr_advanced_tabular_example_resampling_strategy.py` (``example_resampling_strategy.py``) | 09:49.388 | 0.0 MB |
+--------------------------------------------------------------------------------------------------------------------+-----------+--------+
84 changes: 46 additions & 38 deletions ...ctor_development/_sources/basics_tabular/example_tabular_classification.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ with AutoPyTorch

.. code-block:: none
<smac.runhistory.runhistory.RunHistory object at 0x7f2ae1611070> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7fd5a790afd0> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -68,7 +68,7 @@ with AutoPyTorch
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019435882568359375, budget=0), TrajEntry(train_perf=0.17543859649122806, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0014879703521728516, budget=0), TrajEntry(train_perf=0.21052631578947367, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
@@ -100,7 +100,7 @@ with AutoPyTorch
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=6.225258111953735, wallclock_time=8.01650857925415, budget=5.555555555555555), TrajEntry(train_perf=0.14035087719298245, incumbent_id=2, incumbent=Configuration:
, ta_runs=1, ta_time_used=3.994011402130127, wallclock_time=5.4559853076934814, budget=5.555555555555555), TrajEntry(train_perf=0.13450292397660824, incumbent_id=2, incumbent=Configuration:
data_loader:batch_size, Value: 153
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:KernelPCA:gamma, Value: 0.6217858094449208
@@ -133,51 +133,59 @@ with AutoPyTorch
trainer:MixUpTrainer:alpha, Value: 0.7490557199071863
trainer:MixUpTrainer:weighted_loss, Value: False
trainer:__choice__, Value: 'MixUpTrainer'
, ta_runs=4, ta_time_used=53.29574799537659, wallclock_time=62.335920572280884, budget=5.555555555555555), TrajEntry(train_perf=0.17543859649122806, incumbent_id=3, incumbent=Configuration:
data_loader:batch_size, Value: 64
, ta_runs=4, ta_time_used=35.312705755233765, wallclock_time=42.73420023918152, budget=5.555555555555555), TrajEntry(train_perf=0.13450292397660824, incumbent_id=3, incumbent=Configuration:
data_loader:batch_size, Value: 194
encoder:__choice__, Value: 'OneHotEncoder'
feature_preprocessor:__choice__, Value: 'NoFeaturePreprocessor'
imputer:categorical_strategy, Value: 'most_frequent'
imputer:numerical_strategy, Value: 'mean'
lr_scheduler:ReduceLROnPlateau:factor, Value: 0.1
imputer:categorical_strategy, Value: 'constant_!missing!'
imputer:numerical_strategy, Value: 'median'
lr_scheduler:ReduceLROnPlateau:factor, Value: 0.7683488018951772
lr_scheduler:ReduceLROnPlateau:mode, Value: 'min'
lr_scheduler:ReduceLROnPlateau:patience, Value: 10
lr_scheduler:ReduceLROnPlateau:patience, Value: 7
lr_scheduler:__choice__, Value: 'ReduceLROnPlateau'
network_backbone:ShapedMLPBackbone:activation, Value: 'relu'
network_backbone:ShapedMLPBackbone:max_units, Value: 200
network_backbone:ShapedMLPBackbone:mlp_shape, Value: 'funnel'
network_backbone:ShapedMLPBackbone:num_groups, Value: 5
network_backbone:ShapedMLPBackbone:output_dim, Value: 200
network_backbone:ShapedMLPBackbone:activation, Value: 'tanh'
network_backbone:ShapedMLPBackbone:max_units, Value: 316
network_backbone:ShapedMLPBackbone:mlp_shape, Value: 'long_funnel'
network_backbone:ShapedMLPBackbone:num_groups, Value: 6
network_backbone:ShapedMLPBackbone:output_dim, Value: 425
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
network_backbone:__choice__, Value: 'ShapedMLPBackbone'
network_embedding:__choice__, Value: 'NoEmbedding'
network_head:__choice__, Value: 'fully_connected'
network_head:fully_connected:activation, Value: 'relu'
network_head:fully_connected:num_layers, Value: 2
network_head:fully_connected:units_layer_1, Value: 128
network_init:XavierInit:bias_strategy, Value: 'Normal'
network_init:__choice__, Value: 'XavierInit'
optimizer:AdamOptimizer:beta1, Value: 0.9
optimizer:AdamOptimizer:beta2, Value: 0.9
optimizer:AdamOptimizer:lr, Value: 0.01
optimizer:AdamOptimizer:weight_decay, Value: 0.0
optimizer:__choice__, Value: 'AdamOptimizer'
scaler:__choice__, Value: 'StandardScaler'
trainer:StandardTrainer:weighted_loss, Value: True
network_head:fully_connected:units_layer_1, Value: 424
network_init:OrthogonalInit:bias_strategy, Value: 'Zero'
network_init:__choice__, Value: 'OrthogonalInit'
optimizer:RMSpropOptimizer:alpha, Value: 0.6699215268945383
optimizer:RMSpropOptimizer:lr, Value: 0.0009911973694107326
optimizer:RMSpropOptimizer:momentum, Value: 0.11786464509318967
optimizer:RMSpropOptimizer:weight_decay, Value: 0.04607537154099883
optimizer:__choice__, Value: 'RMSpropOptimizer'
scaler:Normalizer:norm, Value: 'max'
scaler:__choice__, Value: 'Normalizer'
trainer:StandardTrainer:weighted_loss, Value: False
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=11, ta_time_used=125.7491409778595, wallclock_time=155.15734958648682, budget=16.666666666666664)]
{'accuracy': 0.8670520231213873}
| | Preprocessing | Estimator | Weight |
|---:|:-----------------------------------------------------|:----------------------------------------------------------------|---------:|
| 0 | None | CatBoostClassifier | 0.48 |
| 1 | None | ExtraTreesClassifier | 0.18 |
| 2 | None | RFClassifier | 0.12 |
| 3 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 4 | None | KNNClassifier | 0.06 |
| 5 | SimpleImputer,NoEncoder,NoScaler,Nystroem | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 6 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 7 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 8 | None | SVC | 0.02 |
, ta_runs=20, ta_time_used=236.7104091644287, wallclock_time=289.09339356422424, budget=50.0)]
{'accuracy': 0.8728323699421965}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
| 0 | None | CatBoostClassifier | 0.14 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 2 | SimpleImputer,OneHotEncoder,Normalizer,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 3 | None | RFClassifier | 0.1 |
| 4 | None | ExtraTreesClassifier | 0.1 |
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 6 | None | SVC | 0.08 |
| 7 | SimpleImputer,NoEncoder,NoScaler,Nystroem | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 8 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 9 | SimpleImputer,OneHotEncoder,NoScaler,NoFeaturePreprocessing | embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 10 | None | KNNClassifier | 0.04 |
| 11 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 12 | SimpleImputer,OneHotEncoder,NoScaler,NoFeaturePreprocessing | embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 13 | SimpleImputer,NoEncoder,MinMaxScaler,TruncSVD | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 14 | None | LGBMClassifier | 0.02 |
| 15 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
@@ -255,7 +263,7 @@ with AutoPyTorch
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 6 minutes 10.960 seconds)
**Total running time of the script:** ( 5 minutes 50.999 seconds)


.. _sphx_glr_download_basics_tabular_example_tabular_classification.py:
12 changes: 6 additions & 6 deletions refactor_development/_sources/basics_tabular/example_tabular_regression.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ with AutoPyTorch

.. code-block:: none
<smac.runhistory.runhistory.RunHistory object at 0x7f2ad93eca90> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<smac.runhistory.runhistory.RunHistory object at 0x7fd58f4c7e20> [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
imputer:categorical_strategy, Value: 'most_frequent'
@@ -66,7 +66,7 @@ with AutoPyTorch
optimizer:__choice__, Value: 'AdamOptimizer'
scaler:__choice__, Value: 'StandardScaler'
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019381046295166016, budget=0), TrajEntry(train_perf=0.37169709500142045, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0013527870178222656, budget=0), TrajEntry(train_perf=0.4696092488249799, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: 'OneHotEncoder'
imputer:categorical_strategy, Value: 'most_frequent'
@@ -96,13 +96,13 @@ with AutoPyTorch
optimizer:__choice__, Value: 'AdamOptimizer'
scaler:__choice__, Value: 'StandardScaler'
trainer:__choice__, Value: 'StandardTrainer'
, ta_runs=1, ta_time_used=4.973564147949219, wallclock_time=9.214993715286255, budget=5.555555555555555)]
{'r2': 0.9023839963347217}
, ta_runs=1, ta_time_used=3.4625701904296875, wallclock_time=6.740808725357056, budget=5.555555555555555)]
{'r2': 0.9158029688846996}
| | Preprocessing | Estimator | Weight |
|---:|:-------------------------------------------|:----------------------------------------------------------------|---------:|
| 0 | SimpleImputer,OneHotEncoder,StandardScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.8 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.16 |
| 2 | SimpleImputer,NoEncoder,NoScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 2 | SimpleImputer,OneHotEncoder,StandardScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
@@ -194,7 +194,7 @@ with AutoPyTorch
.. rst-class:: sphx-glr-timing

**Total running time of the script:** ( 5 minutes 22.534 seconds)
**Total running time of the script:** ( 5 minutes 9.969 seconds)


.. _sphx_glr_download_basics_tabular_example_tabular_regression.py:
6 changes: 3 additions & 3 deletions refactor_development/_sources/basics_tabular/sg_execution_times.rst.txt
View file
Original file line number Diff line number Diff line change
@@ -5,10 +5,10 @@

Computation times
=================
**11:33.494** total execution time for **basics_tabular** files:
**11:00.968** total execution time for **basics_tabular** files:

+----------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_basics_tabular_example_tabular_classification.py` (``example_tabular_classification.py``) | 06:10.960 | 0.0 MB |
| :ref:`sphx_glr_basics_tabular_example_tabular_classification.py` (``example_tabular_classification.py``) | 05:50.999 | 0.0 MB |
+----------------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_basics_tabular_example_tabular_regression.py` (``example_tabular_regression.py``) | 05:22.534 | 0.0 MB |
| :ref:`sphx_glr_basics_tabular_example_tabular_regression.py` (``example_tabular_regression.py``) | 05:09.969 | 0.0 MB |
+----------------------------------------------------------------------------------------------------------+-----------+--------+
2 changes: 1 addition & 1 deletion refactor_development/_static/gallery.css
View file
Original file line number Diff line number Diff line change
@@ -145,7 +145,7 @@ div.sphx-glr-download a:hover {
background-color: #d5d57e;
}

.sphx-glr-example-title > :target::before {
.sphx-glr-example-title:target::before {
display: block;
content: "";
margin-top: -50px;
138 changes: 25 additions & 113 deletions refactor_development/advanced_tabular/example_custom_configuration_space.html
View file
Original file line number Diff line number Diff line change
@@ -143,7 +143,7 @@
</ol>
</div></blockquote>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&lt;smac.runhistory.runhistory.RunHistory object at 0x7f2ad8808a90&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&lt;smac.runhistory.runhistory.RunHistory object at 0x7fd58f4d3d00&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
@@ -177,7 +177,7 @@
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.002296924591064453, budget=0), TrajEntry(train_perf=0.14035087719298245, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001336812973022461, budget=0), TrajEntry(train_perf=0.14619883040935677, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
@@ -211,109 +211,19 @@
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=1, ta_time_used=6.5182719230651855, wallclock_time=8.156251907348633, budget=5.555555555555555), TrajEntry(train_perf=0.14035087719298245, incumbent_id=2, incumbent=Configuration:
data_loader:batch_size, Value: 475
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
imputer:categorical_strategy, Value: &#39;most_frequent&#39;
imputer:numerical_strategy, Value: &#39;constant_zero&#39;
lr_scheduler:__choice__, Value: &#39;NoScheduler&#39;
network_backbone:MLPBackbone:activation, Value: &#39;tanh&#39;
network_backbone:MLPBackbone:dropout_1, Value: 0.579891279191762
network_backbone:MLPBackbone:dropout_2, Value: 0.43202885747368863
network_backbone:MLPBackbone:dropout_3, Value: 0.2053050533304992
network_backbone:MLPBackbone:dropout_4, Value: 0.3628626567848122
network_backbone:MLPBackbone:dropout_5, Value: 0.000687232634536894
network_backbone:MLPBackbone:dropout_6, Value: 0.30779918180581656
network_backbone:MLPBackbone:dropout_7, Value: 0.4566654226669556
network_backbone:MLPBackbone:num_groups, Value: 7
network_backbone:MLPBackbone:num_units_1, Value: 749
network_backbone:MLPBackbone:num_units_2, Value: 751
network_backbone:MLPBackbone:num_units_3, Value: 759
network_backbone:MLPBackbone:num_units_4, Value: 664
network_backbone:MLPBackbone:num_units_5, Value: 219
network_backbone:MLPBackbone:num_units_6, Value: 757
network_backbone:MLPBackbone:num_units_7, Value: 1005
network_backbone:MLPBackbone:use_dropout, Value: True
network_backbone:__choice__, Value: &#39;MLPBackbone&#39;
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.9640640623783606
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.017233504391813814
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.24122690885917664
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.31247176333246596
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.41504826813841933
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.8395119637200936
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.8208414027523236
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.4284420622613293
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 7
network_embedding:__choice__, Value: &#39;LearnedEntityEmbedding&#39;
network_head:__choice__, Value: &#39;fully_connected&#39;
network_head:fully_connected:num_layers, Value: 1
network_init:KaimingInit:bias_strategy, Value: &#39;Zero&#39;
network_init:__choice__, Value: &#39;KaimingInit&#39;
optimizer:AdamOptimizer:beta1, Value: 0.9770847327434384
optimizer:AdamOptimizer:beta2, Value: 0.9710627513919582
optimizer:AdamOptimizer:lr, Value: 0.00010844892447274338
optimizer:AdamOptimizer:weight_decay, Value: 0.05048412416506887
optimizer:__choice__, Value: &#39;AdamOptimizer&#39;
scaler:Normalizer:norm, Value: &#39;max&#39;
scaler:__choice__, Value: &#39;Normalizer&#39;
trainer:StandardTrainer:weighted_loss, Value: False
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=11, ta_time_used=135.23650550842285, wallclock_time=163.7278196811676, budget=16.666666666666664), TrajEntry(train_perf=0.13450292397660824, incumbent_id=3, incumbent=Configuration:
data_loader:batch_size, Value: 174
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:PowerTransformer:standardize, Value: True
feature_preprocessor:__choice__, Value: &#39;PowerTransformer&#39;
imputer:categorical_strategy, Value: &#39;most_frequent&#39;
imputer:numerical_strategy, Value: &#39;most_frequent&#39;
lr_scheduler:ExponentialLR:gamma, Value: 0.9015102188730918
lr_scheduler:__choice__, Value: &#39;ExponentialLR&#39;
network_backbone:MLPBackbone:activation, Value: &#39;relu&#39;
network_backbone:MLPBackbone:num_groups, Value: 2
network_backbone:MLPBackbone:num_units_1, Value: 738
network_backbone:MLPBackbone:num_units_2, Value: 371
network_backbone:MLPBackbone:use_dropout, Value: False
network_backbone:__choice__, Value: &#39;MLPBackbone&#39;
network_embedding:LearnedEntityEmbedding:dimension_reduction_0, Value: 0.5026052624432612
network_embedding:LearnedEntityEmbedding:dimension_reduction_1, Value: 0.7197339745319954
network_embedding:LearnedEntityEmbedding:dimension_reduction_2, Value: 0.9509463920492078
network_embedding:LearnedEntityEmbedding:dimension_reduction_3, Value: 0.5338374300900856
network_embedding:LearnedEntityEmbedding:dimension_reduction_4, Value: 0.2115815132516191
network_embedding:LearnedEntityEmbedding:dimension_reduction_5, Value: 0.3101651859744802
network_embedding:LearnedEntityEmbedding:dimension_reduction_6, Value: 0.724733217572461
network_embedding:LearnedEntityEmbedding:dimension_reduction_7, Value: 0.1459031891772793
network_embedding:LearnedEntityEmbedding:min_unique_values_for_embedding, Value: 4
network_embedding:__choice__, Value: &#39;LearnedEntityEmbedding&#39;
network_head:__choice__, Value: &#39;fully_connected&#39;
network_head:fully_connected:activation, Value: &#39;tanh&#39;
network_head:fully_connected:num_layers, Value: 3
network_head:fully_connected:units_layer_1, Value: 212
network_head:fully_connected:units_layer_2, Value: 414
network_init:NoInit:bias_strategy, Value: &#39;Normal&#39;
network_init:__choice__, Value: &#39;NoInit&#39;
optimizer:RMSpropOptimizer:alpha, Value: 0.5152986504870539
optimizer:RMSpropOptimizer:lr, Value: 0.003121222619830083
optimizer:RMSpropOptimizer:momentum, Value: 0.18725278090773345
optimizer:RMSpropOptimizer:weight_decay, Value: 0.0065085103879551805
optimizer:__choice__, Value: &#39;RMSpropOptimizer&#39;
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: False
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=17, ta_time_used=213.5607807636261, wallclock_time=264.5632119178772, budget=50.0)]
{&#39;accuracy&#39;: 0.8554913294797688}
, ta_runs=1, ta_time_used=4.047108173370361, wallclock_time=5.4710657596588135, budget=5.555555555555555)]
{&#39;accuracy&#39;: 0.8901734104046243}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:----------------------------------------------------------|---------:|
| 0 | None | RFClassifier | 0.26 |
| 1 | None | ExtraTreesClassifier | 0.24 |
| 2 | SimpleImputer,OneHotEncoder,MinMaxScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 3 | None | CatBoostClassifier | 0.1 |
| 4 | None | KNNClassifier | 0.1 |
| 5 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 6 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 7 | SimpleImputer,OneHotEncoder,Normalizer,NoFeaturePreprocessing | embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 8 | None | LGBMClassifier | 0.02 |
| 9 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
&lt;smac.runhistory.runhistory.RunHistory object at 0x7f2ad802a5e0&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
| 0 | None | CatBoostClassifier | 0.28 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.2 |
| 2 | None | ExtraTreesClassifier | 0.18 |
| 3 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.14 |
| 4 | None | KNNClassifier | 0.06 |
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,MLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 6 | SimpleImputer,OneHotEncoder,Normalizer,KernelPCA | embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 7 | None | RFClassifier | 0.04 |
&lt;smac.runhistory.runhistory.RunHistory object at 0x7fd57e5f4d00&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: &#39;NoEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
@@ -345,7 +255,7 @@
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.001631021499633789, budget=0), TrajEntry(train_perf=0.21052631578947367, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0012526512145996094, budget=0), TrajEntry(train_perf=0.19298245614035092, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 32
encoder:__choice__, Value: &#39;NoEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
@@ -377,15 +287,17 @@
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=1, ta_time_used=5.196228265762329, wallclock_time=6.9113147258758545, budget=5.555555555555555)]
, ta_runs=1, ta_time_used=3.548128843307495, wallclock_time=4.968382835388184, budget=5.555555555555555)]
{&#39;accuracy&#39;: 0.8728323699421965}
| | Preprocessing | Estimator | Weight |
|---:|:----------------|:---------------------|---------:|
| 0 | None | ExtraTreesClassifier | 0.34 |
| 1 | None | CatBoostClassifier | 0.28 |
| 2 | None | RFClassifier | 0.2 |
| 3 | None | KNNClassifier | 0.16 |
| 4 | None | SVC | 0.02 |
| | Preprocessing | Estimator | Weight |
|---:|:-------------------------------------------------|:-------------------------------------------------------------------|---------:|
| 0 | None | CatBoostClassifier | 0.24 |
| 1 | None | RFClassifier | 0.22 |
| 2 | None | ExtraTreesClassifier | 0.18 |
| 3 | None | KNNClassifier | 0.14 |
| 4 | None | LGBMClassifier | 0.1 |
| 5 | SimpleImputer,NoEncoder,MinMaxScaler,KitchenSink | no embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 6 | None | SVC | 0.06 |
</pre></div>
</div>
<div class="line-block">
@@ -507,7 +419,7 @@
<span class="nb">print</span><span class="p">(</span><span class="n">api</span><span class="o">.</span><span class="n">show_models</span><span class="p">())</span>
</pre></div>
</div>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 11 minutes 46.927 seconds)</p>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 11 minutes 24.080 seconds)</p>
<div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-advanced-tabular-example-custom-configuration-space-py">
<div class="binder-badge docutils container">
<a class="reference external image-reference" href="https://mybinder.org/v2/gh/automl/Auto-PyTorch/refactor_development?urlpath=lab/tree/notebooks/advanced_tabular/example_custom_configuration_space.ipynb"><img alt="Launch binder" src="../_images/binder_badge_logo.svg" width="150px" /></a>
180 changes: 142 additions & 38 deletions refactor_development/advanced_tabular/example_resampling_strategy.html
View file

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions refactor_development/advanced_tabular/sg_execution_times.html
View file
Original file line number Diff line number Diff line change
@@ -115,7 +115,7 @@

<div class="section" id="computation-times">
<span id="sphx-glr-advanced-tabular-sg-execution-times"></span><h1>Computation times<a class="headerlink" href="#computation-times" title="Permalink to this headline"></a></h1>
<p><strong>22:27.526</strong> total execution time for <strong>advanced_tabular</strong> files:</p>
<p><strong>21:13.468</strong> total execution time for <strong>advanced_tabular</strong> files:</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 86%" />
@@ -124,11 +124,11 @@
</colgroup>
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="example_custom_configuration_space.html#sphx-glr-advanced-tabular-example-custom-configuration-space-py"><span class="std std-ref">Tabular Classification with Custom Configuration Space</span></a> (<code class="docutils literal notranslate"><span class="pre">example_custom_configuration_space.py</span></code>)</p></td>
<td><p>11:46.927</p></td>
<td><p>11:24.080</p></td>
<td><p>0.0 MB</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="example_resampling_strategy.html#sphx-glr-advanced-tabular-example-resampling-strategy-py"><span class="std std-ref">Tabular Classification with different resampling strategy</span></a> (<code class="docutils literal notranslate"><span class="pre">example_resampling_strategy.py</span></code>)</p></td>
<td><p>10:40.599</p></td>
<td><p>09:49.388</p></td>
<td><p>0.0 MB</p></td>
</tr>
</tbody>
84 changes: 46 additions & 38 deletions refactor_development/basics_tabular/example_tabular_classification.html
View file
Original file line number Diff line number Diff line change
@@ -123,7 +123,7 @@
<p>The following example shows how to fit a sample classification model
with AutoPyTorch</p>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&lt;smac.runhistory.runhistory.RunHistory object at 0x7f2ae1611070&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&lt;smac.runhistory.runhistory.RunHistory object at 0x7fd5a790afd0&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
@@ -155,7 +155,7 @@
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019435882568359375, budget=0), TrajEntry(train_perf=0.17543859649122806, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0014879703521728516, budget=0), TrajEntry(train_perf=0.21052631578947367, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
@@ -187,7 +187,7 @@
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=1, ta_time_used=6.225258111953735, wallclock_time=8.01650857925415, budget=5.555555555555555), TrajEntry(train_perf=0.14035087719298245, incumbent_id=2, incumbent=Configuration:
, ta_runs=1, ta_time_used=3.994011402130127, wallclock_time=5.4559853076934814, budget=5.555555555555555), TrajEntry(train_perf=0.13450292397660824, incumbent_id=2, incumbent=Configuration:
data_loader:batch_size, Value: 153
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:KernelPCA:gamma, Value: 0.6217858094449208
@@ -220,51 +220,59 @@
trainer:MixUpTrainer:alpha, Value: 0.7490557199071863
trainer:MixUpTrainer:weighted_loss, Value: False
trainer:__choice__, Value: &#39;MixUpTrainer&#39;
, ta_runs=4, ta_time_used=53.29574799537659, wallclock_time=62.335920572280884, budget=5.555555555555555), TrajEntry(train_perf=0.17543859649122806, incumbent_id=3, incumbent=Configuration:
data_loader:batch_size, Value: 64
, ta_runs=4, ta_time_used=35.312705755233765, wallclock_time=42.73420023918152, budget=5.555555555555555), TrajEntry(train_perf=0.13450292397660824, incumbent_id=3, incumbent=Configuration:
data_loader:batch_size, Value: 194
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
feature_preprocessor:__choice__, Value: &#39;NoFeaturePreprocessor&#39;
imputer:categorical_strategy, Value: &#39;most_frequent&#39;
imputer:numerical_strategy, Value: &#39;mean&#39;
lr_scheduler:ReduceLROnPlateau:factor, Value: 0.1
imputer:categorical_strategy, Value: &#39;constant_!missing!&#39;
imputer:numerical_strategy, Value: &#39;median&#39;
lr_scheduler:ReduceLROnPlateau:factor, Value: 0.7683488018951772
lr_scheduler:ReduceLROnPlateau:mode, Value: &#39;min&#39;
lr_scheduler:ReduceLROnPlateau:patience, Value: 10
lr_scheduler:ReduceLROnPlateau:patience, Value: 7
lr_scheduler:__choice__, Value: &#39;ReduceLROnPlateau&#39;
network_backbone:ShapedMLPBackbone:activation, Value: &#39;relu&#39;
network_backbone:ShapedMLPBackbone:max_units, Value: 200
network_backbone:ShapedMLPBackbone:mlp_shape, Value: &#39;funnel&#39;
network_backbone:ShapedMLPBackbone:num_groups, Value: 5
network_backbone:ShapedMLPBackbone:output_dim, Value: 200
network_backbone:ShapedMLPBackbone:activation, Value: &#39;tanh&#39;
network_backbone:ShapedMLPBackbone:max_units, Value: 316
network_backbone:ShapedMLPBackbone:mlp_shape, Value: &#39;long_funnel&#39;
network_backbone:ShapedMLPBackbone:num_groups, Value: 6
network_backbone:ShapedMLPBackbone:output_dim, Value: 425
network_backbone:ShapedMLPBackbone:use_dropout, Value: False
network_backbone:__choice__, Value: &#39;ShapedMLPBackbone&#39;
network_embedding:__choice__, Value: &#39;NoEmbedding&#39;
network_head:__choice__, Value: &#39;fully_connected&#39;
network_head:fully_connected:activation, Value: &#39;relu&#39;
network_head:fully_connected:num_layers, Value: 2
network_head:fully_connected:units_layer_1, Value: 128
network_init:XavierInit:bias_strategy, Value: &#39;Normal&#39;
network_init:__choice__, Value: &#39;XavierInit&#39;
optimizer:AdamOptimizer:beta1, Value: 0.9
optimizer:AdamOptimizer:beta2, Value: 0.9
optimizer:AdamOptimizer:lr, Value: 0.01
optimizer:AdamOptimizer:weight_decay, Value: 0.0
optimizer:__choice__, Value: &#39;AdamOptimizer&#39;
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:StandardTrainer:weighted_loss, Value: True
network_head:fully_connected:units_layer_1, Value: 424
network_init:OrthogonalInit:bias_strategy, Value: &#39;Zero&#39;
network_init:__choice__, Value: &#39;OrthogonalInit&#39;
optimizer:RMSpropOptimizer:alpha, Value: 0.6699215268945383
optimizer:RMSpropOptimizer:lr, Value: 0.0009911973694107326
optimizer:RMSpropOptimizer:momentum, Value: 0.11786464509318967
optimizer:RMSpropOptimizer:weight_decay, Value: 0.04607537154099883
optimizer:__choice__, Value: &#39;RMSpropOptimizer&#39;
scaler:Normalizer:norm, Value: &#39;max&#39;
scaler:__choice__, Value: &#39;Normalizer&#39;
trainer:StandardTrainer:weighted_loss, Value: False
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=11, ta_time_used=125.7491409778595, wallclock_time=155.15734958648682, budget=16.666666666666664)]
{&#39;accuracy&#39;: 0.8670520231213873}
| | Preprocessing | Estimator | Weight |
|---:|:-----------------------------------------------------|:----------------------------------------------------------------|---------:|
| 0 | None | CatBoostClassifier | 0.48 |
| 1 | None | ExtraTreesClassifier | 0.18 |
| 2 | None | RFClassifier | 0.12 |
| 3 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 4 | None | KNNClassifier | 0.06 |
| 5 | SimpleImputer,NoEncoder,NoScaler,Nystroem | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 6 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 7 | SimpleImputer,NoEncoder,NoScaler,PolynomialFeatures | no embedding,ResNetBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 8 | None | SVC | 0.02 |
, ta_runs=20, ta_time_used=236.7104091644287, wallclock_time=289.09339356422424, budget=50.0)]
{&#39;accuracy&#39;: 0.8728323699421965}
| | Preprocessing | Estimator | Weight |
|---:|:------------------------------------------------------------------|:----------------------------------------------------------------|---------:|
| 0 | None | CatBoostClassifier | 0.14 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 2 | SimpleImputer,OneHotEncoder,Normalizer,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.1 |
| 3 | None | RFClassifier | 0.1 |
| 4 | None | ExtraTreesClassifier | 0.1 |
| 5 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.08 |
| 6 | None | SVC | 0.08 |
| 7 | SimpleImputer,NoEncoder,NoScaler,Nystroem | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 8 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.06 |
| 9 | SimpleImputer,OneHotEncoder,NoScaler,NoFeaturePreprocessing | embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 10 | None | KNNClassifier | 0.04 |
| 11 | SimpleImputer,OneHotEncoder,StandardScaler,KernelPCA | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 12 | SimpleImputer,OneHotEncoder,NoScaler,NoFeaturePreprocessing | embedding,ShapedResNetBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 13 | SimpleImputer,NoEncoder,MinMaxScaler,TruncSVD | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
| 14 | None | LGBMClassifier | 0.02 |
| 15 | SimpleImputer,OneHotEncoder,StandardScaler,NoFeaturePreprocessing | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.02 |
</pre></div>
</div>
<div class="line-block">
@@ -335,7 +343,7 @@
<span class="nb">print</span><span class="p">(</span><span class="n">api</span><span class="o">.</span><span class="n">show_models</span><span class="p">())</span>
</pre></div>
</div>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 6 minutes 10.960 seconds)</p>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 5 minutes 50.999 seconds)</p>
<div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-basics-tabular-example-tabular-classification-py">
<div class="binder-badge docutils container">
<a class="reference external image-reference" href="https://mybinder.org/v2/gh/automl/Auto-PyTorch/refactor_development?urlpath=lab/tree/notebooks/basics_tabular/example_tabular_classification.ipynb"><img alt="Launch binder" src="../_images/binder_badge_logo1.svg" width="150px" /></a>
12 changes: 6 additions & 6 deletions refactor_development/basics_tabular/example_tabular_regression.html
View file
Original file line number Diff line number Diff line change
@@ -123,7 +123,7 @@
<p>The following example shows how to fit a sample regression model
with AutoPyTorch</p>
<p class="sphx-glr-script-out">Out:</p>
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&lt;smac.runhistory.runhistory.RunHistory object at 0x7f2ad93eca90&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
<div class="sphx-glr-script-out highlight-none notranslate"><div class="highlight"><pre><span></span>&lt;smac.runhistory.runhistory.RunHistory object at 0x7fd58f4c7e20&gt; [TrajEntry(train_perf=2147483648, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
imputer:categorical_strategy, Value: &#39;most_frequent&#39;
@@ -153,7 +153,7 @@
optimizer:__choice__, Value: &#39;AdamOptimizer&#39;
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0019381046295166016, budget=0), TrajEntry(train_perf=0.37169709500142045, incumbent_id=1, incumbent=Configuration:
, ta_runs=0, ta_time_used=0.0, wallclock_time=0.0013527870178222656, budget=0), TrajEntry(train_perf=0.4696092488249799, incumbent_id=1, incumbent=Configuration:
data_loader:batch_size, Value: 64
encoder:__choice__, Value: &#39;OneHotEncoder&#39;
imputer:categorical_strategy, Value: &#39;most_frequent&#39;
@@ -183,13 +183,13 @@
optimizer:__choice__, Value: &#39;AdamOptimizer&#39;
scaler:__choice__, Value: &#39;StandardScaler&#39;
trainer:__choice__, Value: &#39;StandardTrainer&#39;
, ta_runs=1, ta_time_used=4.973564147949219, wallclock_time=9.214993715286255, budget=5.555555555555555)]
{&#39;r2&#39;: 0.9023839963347217}
, ta_runs=1, ta_time_used=3.4625701904296875, wallclock_time=6.740808725357056, budget=5.555555555555555)]
{&#39;r2&#39;: 0.9158029688846996}
| | Preprocessing | Estimator | Weight |
|---:|:-------------------------------------------|:----------------------------------------------------------------|---------:|
| 0 | SimpleImputer,OneHotEncoder,StandardScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.8 |
| 1 | SimpleImputer,OneHotEncoder,StandardScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.16 |
| 2 | SimpleImputer,NoEncoder,NoScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
| 2 | SimpleImputer,OneHotEncoder,StandardScaler | no embedding,ShapedMLPBackbone,FullyConnectedHead,nn.Sequential | 0.04 |
</pre></div>
</div>
<div class="line-block">
@@ -274,7 +274,7 @@
<span class="nb">print</span><span class="p">(</span><span class="n">api</span><span class="o">.</span><span class="n">show_models</span><span class="p">())</span>
</pre></div>
</div>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 5 minutes 22.534 seconds)</p>
<p class="sphx-glr-timing"><strong>Total running time of the script:</strong> ( 5 minutes 9.969 seconds)</p>
<div class="sphx-glr-footer class sphx-glr-footer-example docutils container" id="sphx-glr-download-basics-tabular-example-tabular-regression-py">
<div class="binder-badge docutils container">
<a class="reference external image-reference" href="https://mybinder.org/v2/gh/automl/Auto-PyTorch/refactor_development?urlpath=lab/tree/notebooks/basics_tabular/example_tabular_regression.ipynb"><img alt="Launch binder" src="../_images/binder_badge_logo1.svg" width="150px" /></a>
6 changes: 3 additions & 3 deletions refactor_development/basics_tabular/sg_execution_times.html
View file
Original file line number Diff line number Diff line change
@@ -115,7 +115,7 @@

<div class="section" id="computation-times">
<span id="sphx-glr-basics-tabular-sg-execution-times"></span><h1>Computation times<a class="headerlink" href="#computation-times" title="Permalink to this headline"></a></h1>
<p><strong>11:33.494</strong> total execution time for <strong>basics_tabular</strong> files:</p>
<p><strong>11:00.968</strong> total execution time for <strong>basics_tabular</strong> files:</p>
<table class="docutils align-default">
<colgroup>
<col style="width: 85%" />
@@ -124,11 +124,11 @@
</colgroup>
<tbody>
<tr class="row-odd"><td><p><a class="reference internal" href="example_tabular_classification.html#sphx-glr-basics-tabular-example-tabular-classification-py"><span class="std std-ref">Tabular Classification</span></a> (<code class="docutils literal notranslate"><span class="pre">example_tabular_classification.py</span></code>)</p></td>
<td><p>06:10.960</p></td>
<td><p>05:50.999</p></td>
<td><p>0.0 MB</p></td>
</tr>
<tr class="row-even"><td><p><a class="reference internal" href="example_tabular_regression.html#sphx-glr-basics-tabular-example-tabular-regression-py"><span class="std std-ref">Tabular Regression</span></a> (<code class="docutils literal notranslate"><span class="pre">example_tabular_regression.py</span></code>)</p></td>
<td><p>05:22.534</p></td>
<td><p>05:09.969</p></td>
<td><p>0.0 MB</p></td>
</tr>
</tbody>
2 changes: 1 addition & 1 deletion refactor_development/searchindex.js
View file

Large diffs are not rendered by default.

0 comments on commit 6d73691

Please sign in to comment.