Add multi-armed bandit sampler

[ryota717]

Contributor Agreements

Please read the contributor agreements and if you agree, please click the checkbox below.

• I agree to the contributor agreements.

Tip

Please follow the Quick TODO list to smoothly merge your PR.

Motivation

#113

Description of the changes

Adding multi-armed bandit sampler in #113.

TODO List towards PR Merge

Please remove this section if this PR is not an addition of a new package.
Otherwise, please check the following TODO list:

• Copy ./template/ to create your package
• Replace <COPYRIGHT HOLDER> in LICENSE of your package with your name
• Fill out README.md in your package
• Add import statements of your function or class names to be used in __init__.py
• Apply the formatter based on the tips in README.md
• Check whether your module works as intended based on the tips in README.md

Please Check Here

Please tell me if other options(like annealing epsilon, need _n_startup_trials like TPESampler,...) are necessary.

[ryota717]

[QUESTION] This defaultdict treats never choiced arm having 0 rewards. Should i replace any other idea?
(as far as I can think of using _n_startup_trials like TPESampler or letting user set default reward instead of 0)

[nabenabe0928]

That is a very good point actually:)
I should have given the pseudocode of the $\epsilon$-greedy algorithm, but it usually works as follows:

1. The control parameter of the algorithm is $\epsilon$, i.e. the probability of random sampling, n_trials, which we define as $T$ hereafter, and the number of choices $K$.
2. Try every single arm $\epsilon T / K$ times.
3. Choose the optimal arm (up to $\epsilon T / K$ or up to the latest trial) for each dimension.

So usually, we start from the random initialization.
However, we do not have to strictly stick to this algorithm, meaning that it is totally acceptable to not follow the classic algorithm implementation.
Instead, we can do it in the UCB policy fashion where we try each arm once at the initialization.
In this way, your issue will be resolved and we can still retain most of your implementation.

[ryota717]

Thanks for your suggestion!

Instead, we can do it in the UCB policy fashion where we try each arm once at the initialization.

This looks me to good and changed initialization in 371556f
(random initialization seems difficult for Optuna because of its high objective flexibility 🙏)

[y0z]

@nabenabe0928

Could you review this PR? (cf. #113 (comment))

[nabenabe0928]

I confirmed that the example works!

[nabenabe0928]

That is a very good point actually:)
I should have given the pseudocode of the $\epsilon$-greedy algorithm, but it usually works as follows:

1. The control parameter of the algorithm is $\epsilon$, i.e. the probability of random sampling, n_trials, which we define as $T$ hereafter, and the number of choices $K$.
2. Try every single arm $\epsilon T / K$ times.
3. Choose the optimal arm (up to $\epsilon T / K$ or up to the latest trial) for each dimension.

So usually, we start from the random initialization.
However, we do not have to strictly stick to this algorithm, meaning that it is totally acceptable to not follow the classic algorithm implementation.
Instead, we can do it in the UCB policy fashion where we try each arm once at the initialization.
In this way, your issue will be resolved and we can still retain most of your implementation.

[ryota717]

@nabenabe0928 Thank you for suggestion and comment! Colud you check my revisions, please?

[ryota717]

Thanks for your suggestion!

Instead, we can do it in the UCB policy fashion where we try each arm once at the initialization.

This looks me to good and changed initialization in 371556f
(random initialization seems difficult for Optuna because of its high objective flexibility 🙏)

[nabenabe0928]

@ryota717

Hi, thank you for the prompt action!
I will look into the changes asap:)
But probably, it is better to rename the directory name to something like mab_epsilon_greedy?

[ryota717]

@nabenabe0928 Renamed modules and directory in 0bc6cb7.

[nabenabe0928]

@ryota717
Thank you so much! I will check the rest asap 🙇

[nabenabe0928]

Thank you for the modification and sorry for the late response:(
I added some comments, but you can choose whether you take the suggestions or not!
Feel free to tell me your opinion and then we can promptly merge this PR!

[nabenabe0928]

[nit]

Suggested change

	title: MAB Epsilon-Greedy Sampler
	title: A Sampler Based on Epsilon-Greedy Multi-Armed Bandit Algorithm

[nabenabe0928]

[nit]
Now, thanks to the last modification, we do not have to use min/max operator here!

Suggested change

	key=lambda x: rewards_by_choice[x] / max(cnt_by_choice[x], 1),
	key=lambda x: rewards_by_choice[x] / cnt_by_choice[x],

[nabenabe0928]

[nit]
Same here:)

Suggested change

	key=lambda x: rewards_by_choice[x] / max(cnt_by_choice[x], 1),
	key=lambda x: rewards_by_choice[x] / cnt_by_choice[x],

[nabenabe0928]

I will merge this PR once, so if you would like to add my suggestions, please publish another PR!