A single ab_test call triggers multiple Redis commands for all experiments

[phstc]

Describe the bug

ab_test triggers multiple Redis commands for all experiments for a single experiment ab_test(:experiment_1) call.

If in a single request flow I call ab_test(:experiment_1), ab_test(:experiment_2) and ab_test(:experiment_3) it will perform multiple commands for the same experiments overloading Redis.

To Reproduce

Open redis-cli monitor then:

ab_test(:experiment_1)

A single call will perform get, hget, type, lrange etc for all other experiments defined in the experiments.yml.

Expected behavior

Redis calls only for the given experiment.

[andrehjr]

Hi @phstc

Thanks for reporting. Yes, it's a known issue, the latest master branch already reduced a lot of calls. But there is still a lot more improvements to go.

I'll release a new version that partially addresses this issue.

[phstc]

Hi @andrehjr

Thank you for the prompt answer.

Is master stable?

I'll release a new version that partially addresses this issue.

❤️ ⭐️ 👏

[phstc]

👋 @andrehjr

The lastest version decreased the number of Redis calls from ~3,600 per page to just ~240. Great work, thank you.

We track Redis response times in New Relic, dividing the response time per number of active experiments, we got ~4.5m per experiment. Can we take that as a solid reference for the time spent per experiment or does that vary based on experiments?

[andrehjr]

HI @phstc! Awesome!

If I understood it correctly, I believe that's an accurate number. Do you have more than one experiment per page? How many active experiments do you have?

On the current split version, the total of Redis calls depends directly on the number of active_experiments for a given user. That's something I'm working on and should reduce the majority of the Redis calls that you're seeing.

[phstc]

Do you have more than one experiment per page? How many active experiments do you have?

We have a SPA, so for the first-page load, we load all experiments and keep them in the JS scope.

How many active experiments do you have?

It can vary. That's why we want to know how long each experiment would add to the response times.

With the current number of active experiments, we see an AVG of 4.5 ms per experiment.

I was wondering if that 4.5 ms could be used as a reference.

Is the time spent more linear, or could it be exponential based on the number of experiments?

[andrehjr]

I see! Thanks for the extra details.

When we call ab_test, we loop through the user's active experiments and remove all the finished experiments. It's helpful for housekeeping and cleaning up old experiments data. But it is not good performance-wise when you have a lot of active experiments in the way that it's currently implemented.

It's done once per request/user. That's probably the root of the majority of the extra calls you see on NewRelic.

The time spent is more linear and not exponential. I hope to have more news on this soon :)