-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi arm bandit or A/B testing? #3
Comments
@robertleeplummerjr - yes, we built it as a proof of concept on a private repo of Mojito JS Delivery. We aren't really using MAB tests for our clients because we find it's easier for us to report on straight traffic splits and check for SRM. But if it's of interest to the community, we'd be happy to build this out a bit further and share how it works. At a high-level, our proof of concept works by sharing the test results with
The complexity lies in:
It's not an out-of-the-box solution either. You'd need to have the chops to expose your tests results to Mojito's CI pipeline. Though it only took us a few hours to script this up (with intimate knowledge): |
I feel compelled to point out that the stationarity assumption, which is required by many MAB algorithms (such as the popular Thompson sampling bandit), hardly ever holds in online practice. Violation of this assumption makes the bandit susceptible to Simpson's paradox, resulting in the bandit converging on suboptimal arms in practice more often than the theory suggests. |
Oh, and technically speaking, AB testing is already a solution to the MAB problem. :-D It's called "epsilon first". We first spend a fixed amount of time/sample (i.e. epsilon) fully exploring (e.g. randomly sampling arms) and then commit to fully exploiting the most likely optimal arm for the remainder of the sample (e.g. we ship the winning variation). In cases where the sample is not limited (e.g. there is no fixed limit or budget to the sample) this solution can actually be quite efficient. |
I was curious if this was a feature set that is either supported, or planned to be supported in the future.
The text was updated successfully, but these errors were encountered: