[bug] Using Priors causes half of runs to return infinity

[Dvermetten]

Description

I want to make use of priors in my SMAC runs, but whenever I add a parameter with a prior to my configspace, it results in half of my trials getting an infinite cost value since they don't finish running.

Steps/Code to Reproduce

I took the configspace example from https://automl.github.io/ConfigSpace/latest/guide/#5th-example-placing-priors-on-the-hyperparameters, added a fake evaluation function with a simple HPOInterface:

from ConfigSpace import Configuration, ConfigurationSpace
from ConfigSpace import Normal, Integer, Float, Beta, Categorical

from smac import Scenario, HyperparameterOptimizationFacade

from itertools import product

import numpy as np
import time


cs = ConfigurationSpace(
    space={
        "lr": Float(
            'lr',
            bounds=(1e-5, 1e-1),
            default=1e-3,
            log=True,
            distribution=Normal(1e-3, 1e-1)
        ),
        "dropout": Float(
            'dropout',
            bounds=(0, 0.99),
            default=0.25,
            distribution=Beta(alpha=2, beta=4)
        ),
        "activation": Categorical(
            'activation',
            items=['tanh', 'relu'],
            weights=[0.2, 0.8]
        ),
    },
    seed=1234,
)


def get_fake_performance(config : Configuration, instance: str, seed: int = 0):
    return [np.random.uniform()]

def run_smac(fids):
    inst_feats = {str(arg): [idx] for idx, arg in enumerate(fids)}
    scenario = Scenario(
        cs,
        name=str(int(time.time())) + "-" + "example",
        deterministic=False,
        min_budget=2,
        max_budget=20,
        n_trials=200,
        instances=fids,
        instance_features=inst_feats,
        output_directory="smac3_output", 
        n_workers=2
    )
    smac = HyperparameterOptimizationFacade(scenario, get_fake_performance)
    smac.optimize()

run_smac([1,2,3])

The same occurs when using the example from https://automl.github.io/SMAC3/main/examples/1_basics/6_priors.html and adding n_workers=2 to the scenario (and replacing the evaluation function with a random number to save time testing).

Snippet from a runhistory file to illustrate:

"stats": {
    "submitted": 42,
    "finished": 21,
    "running": 21
  },

Expected Results

A normal SMAC run

Actual Results

A run in which half of the configurations don't finish running / time out (even though there is no evaluation function which takes time).

Versions

Tested on Python 3.10 and 3.11.2, with smac==2.3.0 and configspace==1.2.1

[LukasFehring]

Hi Diederick,
I am now trying to investigate this issue

First and as a side note: to use Priors properly, you need to use the PriorAcquisitionFunction as given in the example. Your provided code block does not do that, the quoted example does.

PriorAcquisitionFunction(
        acquisition_function=HyperparameterOptimizationFacade.get_acquisition_function(scenario),
        decay_beta=scenario.n_trials / 10,  # Proven solid value
    )

Secondly, I agree that this is a confusing issue. I think this is likely connected to DASK because my execution of your code also ends in a DASK error. How critical is it for you to use multiple workers? We are not sure whether there is a simple fix for this issue.