Refactor: Documentation adapted to SklearnOptimizer and Hyperparamete…

…rSpace classes.

README.md

@@ -10,6 +10,7 @@
 ## Features
 - Easy to use
 - DEAP-based genetic algorithm ready to use with several machine learning algorithms
+- Adaptable to use with any machine learning algorithm that complies with the Scikit-Learn API
 - Default hyperparameter ranges
 - Default score functions for evaluating the performance of the model
 - Reproducibility of results
@@ -47,18 +48,24 @@ You can get more information about the package installation at https://pypi.org/
 Here's a simple example of how to optimize hyperparameters in a decision tree classifier using the iris dataset:
 
 ```python
-from mloptimizer.genoptimizer import TreeOptimizer
+from mloptimizer.genoptimizer import SklearnOptimizer
+from mloptimizer.hyperparams import HyperparameterSpace
+from sklearn.tree import DecisionTreeClassifier
 from sklearn.datasets import load_iris
 
+# 1) Load the dataset and get the features and target
 X, y = load_iris(return_X_y=True)
 
-#   
-opt = TreeOptimizer(X, y, "Optimizer")
+# 2) Define the hyperparameter space (a default space is provided for some algorithms)
+hyperparameter_space = HyperparameterSpace.get_default_hyperparameter_space(DecisionTreeClassifier)
+
+# 3) Create the optimizer and optimize the classifier
+opt = SklearnOptimizer(clf_class=DecisionTreeClassifier, features=X, labels=y, hyperparam_space=hyperparameter_space)
 
 clf = opt.optimize_clf(10, 10)
 ```
 
-The last line of code will create a directory in the current folder with a name like `YYYYMMDD_nnnnnnnnnn_TreeOptimizer`.
+The last line of code will create a directory in the current folder with a name like `YYYYMMDD_nnnnnnnnnn_SklearnOptimizer`.
 This folder contains the results of the optimization, 
 including the best estimator found and the log file `opt.log` informing with all the steps, 
 the best estimator and the result of the optimization.

docs/sections/Basics/overview.rst

@@ -4,26 +4,33 @@ Overview
 
 Introduction
 ------------
-Every subclass of BaseOptimizer should be used in a similar way. As example we are taking the Optimizer
-for the XGBoost Classifier, the `XGBClassifierOptimizer`.
+The main class objects are the `SklearnOptimizer` and the `HyperparameterSpace` classes.
+
+The optimizer `SklearnOptimizer` is able to optimize any model that complies with the `sklearn` API.
+The `HyperparameterSpace` class is used to define the hyperparameters that will be optimized, either
+the fixed hyperparameters or the hyperparameters that will be optimized.
 
 Usage
 -----
-To use the `XGBClassifierOptimizer` class:
+To use the `SklearnOptimizer` class:
 
 1. Define your features and labels.
-2. Create an instance of `XGBClassifierOptimizer` with your data and any optional parameters.
+2. Choose a model to optimize that complies with the `sklearn` API. (e.g. `XGBClassifier`).
+2. Create an instance of `HyperparameterSpace` with the hyperparameters that you want to optimize.
 3. Call the `optimize_clf()` method to start the optimization process.
 
-There are several parameters than can be passed to the `XGBClassifierOptimizer`
-(or any subclass of `BaseOptimizer`) constructor:
+.. note::
+    There are default HyperparameterSpaces defined in the ``conf`` folder for the most common models.
+    You can use the HyperparameterSpace.get_default_hyperparams(class) (class e.g. XGBClassifier).
+
+There are several parameters than can be passed to the `SklearnOptimizer` constructor:
 
+- `clf_class`: The class of the model to optimize. It should comply with the `sklearn` API.
 - `X`: The features of your dataset.
 - `y`: The labels of your dataset.
 - `folder`: The folder where the files and folder will be saved. Defaults to the current directory.
 - `log_file`: The name of the log file. Defaults to `mloptimizer.log`.
-- `custom_hyperparams`: A dictionary of custom hyperparameters. The key of each hyperparameter is the name of the hyperparameter, and the value is the `Hyperparam` object itself. The `Hyperparam` object constructor takes the following parameters:
-- `fixed_hyperparams`: A dictionary of fixed hyperparameters. The key of each hyperparameter is the name of the hyperparameter, and the value is the value of the hyperparameter.
+- `hyperparam_space`: The hyperparameter space to use for the optimization process.
 - `eval_function`: The function to use to evaluate the model. Defaults to `train_score`.
 - `score_function`: The function to use to score the model. Defaults to `accuracy_score`.
 - `seed`: The seed to use for reproducibility. Defaults to a random integer between 0 and 1000000.
@@ -35,79 +42,82 @@ Default Usage Example
 The simplest example of using the Optimizer is:
 
 - Store your features and labels in `X` and `y` respectively.
-- Create an instance of `XGBClassifierOptimizer` with your data and leave all other parameters to their default values.
+- Use HyperparameterSpace.get_default_hyperparams(XGBClassifier) to get the default hyperparameters for the model you want to optimize.
+- Create an instance of `SklearnOptimizer` with your classifier class, hyperparameter space, data and leave all other parameters to their default values.
 - Call the `optimize_clf()` method to start the optimization process. You can pass the population size and the number of generations to the method.
 - The result of the optimization process will be a object of type XGBClassifier with the best hyperparameters found.
 
 .. code-block:: python
 
-    # Load your data
-    X, y = load_your_data()
+    from mloptimizer.genoptimizer import SklearnOptimizer
+    from mloptimizer.hyperparams import HyperparameterSpace
+    from xgboost import XGBClassifier
+    from sklearn.datasets import load_iris
 
-    # Create an instance of XGBClassifierOptimizer
-    xgb_optimizer = XGBClassifierOptimizer(X, y)
+    # 1) Load the dataset and get the features and target
+    X, y = load_iris(return_X_y=True)
 
-    # Start the optimization process
-    result = xgb_optimizer.optimize_clf(3, 3)
+    # 2) Define the hyperparameter space (a default space is provided for some algorithms)
+    hyperparameter_space = HyperparameterSpace.get_default_hyperparameter_space(XGBClassifier)
 
-This will create a folder (in the current location) with name `YYYYMMDD_nnnnnnnnnn_XGBClassifierOptimizer`
+    # 3) Create the optimizer and optimize the classifier
+    opt = SklearnOptimizer(clf_class=XGBClassifier, features=X, labels=y, hyperparam_space=hyperparameter_space)
+
+    clf = opt.optimize_clf(10, 10)
+
+This will create a folder (in the current location) with name `YYYYMMDD_nnnnnnnnnn_SklearnOptimizer`
 (where `YYYYMMDD_nnnnnnnnnn` is the current timestamp) and a log file named `mloptimizer.log`.
 To inspect the structure of the folder and what can you find in it, please refer to the `Folder Structure` section.
 
-Custom Hyperparameters Example
-------------------------------
+Custom HyperparameterSpace Example
+----------------------------------
 
-Among the parameters that can be passed to the `XGBClassifierOptimizer` (or other Optimizer) constructor,
-there are two that are used to define custom hyperparameters: `custom_hyperparams` and `fixed_hyperparams`.
+Among the parameters that can be passed to the `SklearnOptimizer` constructor,
+the `hyperaram_space` of class `HyperparameterSpace` is really important
+and should be aligned with the machine learning algorithm passed to the Optimizer: `fixed_hyperparams`
+and `evolvable_hyperparams`.
 
-The `custom_hyperparams` parameter is a dictionary of custom hyperparameters.
+The `evolvable_hyperparams` parameter is a dictionary of custom hyperparameters.
 The key of each hyperparameter is the name of the hyperparameter, and the value is the `Hyperparam` object itself.
 To understand how to use the `Hyperparam` object, please refer to the `Hyperparam` section inside Concepts.
 
+The `fixed_hyperparams` parameter is a dictionary of fixed hyperparameters.
+This is simply a dictionary where the key is the name of the hyperparameter, and the value is the value of the hyperparameter.
+These hyperparameters will not be optimized, but will be used as fixed values during the optimization process.
+
 An example of using custom hyperparameters is:
 
 .. code-block:: python
 
     # Define your custom hyperparameters
-    custom_hyperparams = {
-        'colsample_bytree': Hyperparam("colsample_bytree", 3, 10, float, 10),
-        'gamma': Hyperparam("gamma", 0, 20, int),
-        'learning_rate': Hyperparam("learning_rate", 1, 100, float, 1000),
-        'max_depth': Hyperparam("max_depth", 3, 20, int),
-        'n_estimators': Hyperparam("n_estimators", 100, 500, int),
-        'subsample': Hyperparam("subsample", 700, 1000, float, 1000),
-        'scale_pos_weight': Hyperparam("scale_pos_weight", 15, 40, float, 100)
+    fixed_hyperparams = {
+        'max_depth': 5
+    }
+    evolvable_hyperparams = {
+        'colsample_bytree': Hyperparam("colsample_bytree", 3, 10, 'float', 10),
+        'gamma': Hyperparam("gamma", 0, 20, 'int'),
+        'learning_rate': Hyperparam("learning_rate", 1, 100, 'float', 1000),
+        # 'max_depth': Hyperparam("max_depth", 3, 20, 'int'),
+        'n_estimators': Hyperparam("n_estimators", 100, 500, 'int'),
+        'subsample': Hyperparam("subsample", 700, 1000, 'float', 1000),
+        'scale_pos_weight': Hyperparam("scale_pos_weight", 15, 40, 'float', 100)
     }
 
+
+    custom_hyperparam_space = HyperparameterSpace(fixed_hyperparams, evolvable_hyperparams)
+
     # Create an instance of XGBClassifierOptimizer with custom hyperparameters
-    xgb_optimizer = XGBClassifierOptimizer(X, y, hyperparams=custom_hyperparams)
+    xgb_optimizer = SklearnOptimizer(clf_class=XGBClassifier,features=X, labels=y,
+                                     hyperparam_space=custom_hyperparam_space)
 
     # Start the optimization process
     result = xgb_optimizer.optimize_clf(3, 3)
 
 
-The `fixed_hyperparams` parameter is a dictionary of fixed hyperparameters.
-This is simply a dictionary where the key is the name of the hyperparameter, and the value is the value of the hyperparameter.
-These hyperparameters will not be optimized, but will be used as fixed values during the optimization process.
-
-An example of using fixed hyperparameters is:
-
-.. code-block:: python
-
-    # Define your fixed parameters
-    fixed_params = {
-        'n_estimators': 100,
-        'max_depth': 5
-    }
-
-    # Create an instance of XGBClassifierOptimizer with fixed parameters
-    xgb_optimizer = XGBClassifierOptimizer(X, y, fixed_hyperparams=fixed_params)
 
-    # Start the optimization process
-    result = xgb_optimizer.optimize_clf(3, 3)
 
 
-Both `custom_hyperparams` and `fixed_hyperparams` can be used together,
+Both `evolvable_hyperparams` and `fixed_hyperparams` can be used together,
 providing several different ways to customize the optimization process.
 
 Reproducibility
@@ -117,20 +127,33 @@ Researchers often need to be able to reproduce their results. During the researc
 advisable to run several optimizations processes with different parameters or input data.
 However, if the results of the optimization process are not reproducible, it will be difficult to compare
 the results of the different optimization processes.
-In order to make the results reproducible, the `XGBClassifierOptimizer` (and all other Optimizers) have a `seed` parameter.
+In order to make the results reproducible, the `SklearnOptimizer` have a `seed` parameter.
 This parameter is used to set the seed of the random number generator used during the optimization process.
 If you set the same seed, the results of the optimization process will be the same.
 
 An example of two executions of the optimization process with the same seed that will produce the same result is:
 
 .. code-block:: python
 
-    # Create two instances of XGBClassifierOptimizer with the same seed
-    xgb_optimizer1 = XGBClassifierOptimizer(X, y, seed=42)
-    result1 = xgb_optimizer1.optimize(3, 3)
+    from mloptimizer.genoptimizer import SklearnOptimizer
+    from mloptimizer.hyperparams import HyperparameterSpace
+    from xgboost import XGBClassifier
+    from sklearn.datasets import load_iris
+
+    # 1) Load the dataset and get the features and target
+    X, y = load_iris(return_X_y=True)
+
+    # 2) Define the hyperparameter space (a default space is provided for some algorithms)
+    hyperparameter_space = HyperparameterSpace.get_default_hyperparameter_space(XGBClassifier)
+
+    # 3) Create two instances of SklearnOptimizer with the same seed
+    xgb_optimizer1 = SklearnOptimizer(clf_class=XGBClassifier, features=X, labels=y,
+                                      hyperparam_space = hyperparameter_space, seed=42)
+    result1 = xgb_optimizer1.optimize_clf(3, 3)
 
-    xgb_optimizer2 = XGBClassifierOptimizer(X, y, seed=42)
-    result2 = xgb_optimizer2.optimize(3, 3)
+    xgb_optimizer2 = SklearnOptimizer(clf_class=XGBClassifier, features=X, labels=y,
+                                      hyperparam_space = hyperparameter_space, seed=42)
+    result2 = xgb_optimizer2.optimize_clf(3, 3)
 
     # Verify that the results are the same
     # The comparison is done using the string representation of the result objects

docs/sections/Concepts/hyperparam.rst

@@ -31,15 +31,15 @@ The `Hyperparam` class supports several types of hyperparameters. Here are examp
 .. code-block:: python
 
    hyperparam_int = Hyperparam(name='max_depth', min_value=1,
-                               max_value=10, hyperparam_type=int)
+                               max_value=10, hyperparam_type='int')
 
 
 - Float hyperparameter:
 
 .. code-block:: python
 
     hyperparam_float = Hyperparam(name='learning_rate', min_value=0.01, max_value=1.0,
-                                  hyperparam_type=float, denominator=100)
+                                  hyperparam_type='float', scale=100)
 
 
 - 'nexp' hyperparameter:
@@ -67,7 +67,7 @@ Here's an example of how to use the Hyperparam class:
 
    # Define a hyperparameter
    hyperparam = Hyperparam(name='learning_rate', min_value=0, max_value=1,
-                           hyperparam_type=float, denominator=100)
+                           hyperparam_type='float', scale=100)
 
    # Correct a value
    # This will return 1.0 as 150 is beyond the max_value
@@ -76,28 +76,40 @@ Here's an example of how to use the Hyperparam class:
 
 In this example, we define a hyperparameter named 'learning_rate' with a minimum value of 0, a maximum value of 1, and a type of float. The 'correct' method is then used to correct a value that is beyond the defined maximum value.
 
-Here's an example of how you can create a `TreeOptimizer` instance and pass custom hyperparameters to it:
+Here's an example of how you can create a `HyperparameterSpace` instance and pass custom hyperparameters to it:
 
 .. code-block:: python
 
-   from mloptimizer.genoptimizer import Hyperparam
-   from mloptimizer.genoptimizer.trees import TreeOptimizer
+   from mloptimizer.hyperparams import Hyperparam, HyperparameterSpace
 
    # Define custom hyperparameters
-   custom_hyperparams = {
-   "min_samples_split": Hyperparam("min_samples_split", 2, 50, int),
-   "min_samples_leaf": Hyperparam("min_samples_leaf", 1, 20, int),
-   "max_depth": Hyperparam("max_depth", 2, 20, int),
-   "min_impurity_decrease": Hyperparam("min_impurity_decrease", 0, 150, float, 1000),
-   "ccp_alpha": Hyperparam("ccp_alpha", 0, 300, float, 100000)
+   fixed_hyperparams = {
+       "criterion": "gini"
    }
+   evolvable_hyperparams = {
+       "min_samples_split": Hyperparam("min_samples_split", 2, 50, 'int'),
+       "min_samples_leaf": Hyperparam("min_samples_leaf", 1, 20, 'int'),
+       "max_depth": Hyperparam("max_depth", 2, 20, 'int'),
+       "min_impurity_decrease": Hyperparam("min_impurity_decrease", 0, 150, 'float', 1000),
+       "ccp_alpha": Hyperparam("ccp_alpha", 0, 300, 'float', 100000)
+   }
+
+
+   # Create a HyperparameterSpace instance
+   hyperparam_space = HyperparameterSpace(fixed_hyperparams, evolvable_hyperparams)
+
+   # Then we can use the hyperparam_space instance to optimize the hyperparameters
+   from sklearn.tree import DecisionTreeClassifier
+   from sklearn.datasets import load_iris
+   from mloptimizer.genoptimizer import SklearnOptimizer
 
-   # Create a TreeOptimizer instance
-   tree_optimizer = TreeOptimizer()
+   # Load the iris dataset
+   X,y = load_iris(return_X_y=True)
 
-   # Set the custom hyperparameters
-   tree_optimizer.hyperparams = custom_hyperparams
+   tree_optimizer = SklearnOptimizer(clf_class=DecisionTreeClassifier,
+                                    hyperparam_space=hyperparam_space,
+                                    features=X, labels=y)
+   tree_optimizer.optimize_clf(3, 3)
 
-In this example, we first define a dictionary of custom hyperparameters. Each hyperparameter is an instance of the `Hyperparam` class, which takes the name, minimum value, maximum value, type, and an optional denominator as arguments.
 
-Then, we create an instance of the `TreeOptimizer` class and set its `hyperparams` attribute to our custom hyperparameters.
+In this example, we define custom hyperparameters and create a `HyperparameterSpace` instance. We then use the `HyperparameterSpace` instance to optimize the hyperparameters of a `DecisionTreeClassifier` using the `SklearnOptimizer` class.

docs/sections/Concepts/parallel.rst

@@ -14,28 +14,36 @@ An example of the speedup that can be achieved using parallel processing is show
    In the example below, the seed is set to 25 to ensure the result using parallel processing is the same as the one without parallel processing.
 
 .. warning::
-   Parallel processing is not supported for the ``XGB`` and ``Keras`` classifiers.
+   Parallel processing is not supported for the ``XGBClassifier`` and ``KerasClassifier`` classifiers.
 
 .. code-block:: python
 
+    from mloptimizer.genoptimizer import SklearnOptimizer
+    from mloptimizer.hyperparams import HyperparameterSpace
+    from sklearn.tree import DecisionTreeClassifier
     from sklearn.datasets import load_iris
     import time
 
-    from mloptimizer.genoptimizer import TreeOptimizer
+    # Load the dataset and get the features and target
+    X, y = load_iris(return_X_y=True)
 
-    X, y = dataset(return_X_y=True)
+    # Define the hyperparameter space (a default space is provided for some algorithms)
+    hyperparameter_space = HyperparameterSpace.get_default_hyperparameter_space(DecisionTreeClassifier)
+
+    # Set the seed to ensure the result using parallel processing is the same as the one without parallel processing
     my_seed = 25
     population = 50
     generations = 4
 
-    opt_with_parallel = optimizer(X, y, seed=my_seed, use_parallel=True)
+    opt_with_parallel = SklearnOptimizer(clf_class=DecisionTreeClassifier, features=X, labels=y,
+                                  hyperparam_space=hyperparameter_space, seed=my_seed, use_parallel=True)
 
     start_time_parallel = time.time()
     clf_with_parallel = opt_with_parallel.optimize_clf(population, generations)
     end_time_parallel = time.time()
 
-    opt = optimizer(X, y, seed=my_seed, use_parallel=False)
-
+    opt = SklearnOptimizer(clf_class=DecisionTreeClassifier, features=X, labels=y,
+                    hyperparam_space=hyperparameter_space, seed=my_seed, use_parallel=False)
     start_time = time.time()
     clf = opt.optimize_clf(population, generations)
     end_time = time.time()