Not using all instances?

[ketildanielsen]

After 5000 of n_trials=20000, it has used only 3 of the 18 instances provided. My "scenario.json" confirms all 18 are in the "instances" part, each having 3 features in the "instance_features" part.

Everything else seems to run very well, but I want SMAC to expose the target algorithm to all 18 instances!

Any parameter I should use? Appreciate any advice. The code is a modification of one of the examples:

if __name__ == "__main__":
    model = MySearch ()
    scenario = Scenario(model.configspace,
        deterministic=False,  
        instances=instance_filenames,   # list of 18 file names 
        instance_features=instance_features,   # dict 18 items, each a list of 3 integer features
        seed=-1, 
        n_workers=20, 
        n_trials=20000)

    intensifier = HyperparameterOptimizationFacade.get_intensifier(
        scenario,
        max_config_calls=1,  # We basically use one seed per config only
    )

    smac = HyperparameterOptimizationFacade(
        scenario,
        model.train,
        intensifier=None, 
        overwrite=True,
        logging_level=20,
    )

    incumbent = smac.optimize()

[ketildanielsen]

Update: By setting intensifier=intensifier and max_config_calls to 18, they (all 18) are now used. Very unevenly, not round robin order. In the beginning, it seems to select 5-6 most of the time, then (slowly) the rest are brought in. Anyway, an improvement.

[benjamc]

Hi ketildanielsen,
this is due to the intensification process. During that process we need to estimate whether a new, challenging configuration (called challenger) performs better than the current best incumbent configuration. In order to become the new best, the challenger needs to be at least as good and better on all instances the incumbent is evaluated on.
If we would evaluate each configuration on all instances we would waste evaluations for unpromising challengers. The intensification therefore aims to distribute the compute budget such that new configurations can be tried and configurations are evaluated on many instances.
You can read further about it in the original SMAC paper in section 3.
If you have any further questions, please let us know.

[ketildanielsen]

Thanks.

I also wondered about this behavior: In a very small case with n_trials=30 and parallel=2, it completes the 30 using 2 cores, only using 8-9 of the 18 instances. Then it continues to do 40 runs sequentially, eventually using all 18 instances. See plot (below). It then prints out the incumbent. Eventually it times out on an error:

Exception ignored in: <function DaskParallelRunner.__del__ at 0x7fd28afd9b20>
Traceback (most recent call last):
  File "/mnt/c/users/k/smac-env/lib/python3.12/site-packages/smac/runner/dask_runner.py", line 206, in __del__
  File "/mnt/c/users/k/smac-env/lib/python3.12/site-packages/smac/runner/dask_runner.py", line 180, in close
  File "/mnt/c/users/k/smac-env/lib/python3.12/site-packages/distributed/client.py", line 1997, in close
  File "/mnt/c/users/k/smac-env/lib/python3.12/site-packages/distributed/utils.py", line 433, in sync
TimeoutError: timed out after 60 s.

The redline is before (30) and after (40).

I was happy I got an incumbent. But, this error message worries me. There are 30 entries in the runhistory.json file. The 40 additional runs not mentioned.