Improve load balancing of App Proxy

[achimnol]

Currently we are using a uniform random variable to choose a route from the weighted route list.
The current implementation is a our own Python logic, but this could be optimized by replacing it with random.choices() which is provided as a native C implementation in CPython.

Also, let's add an option to switch the load balancing algorithm to introduce a proper weighted round robin with shuffling as follows:

weights = {
  a: 2.5,
  b: 1.2,
  c: 1.0,
}
multiplier = 10
shuffle_period = 5
route_decisions = [
  route * int(weights[r] * multiplier) for route, weight in routes.items()
]
random.shuffle(route_decisions)

current_choice = 0
pass_count = 0

def select_route():
  global current_choice, pass_count
  selected_route = route_decisions[current_choice]
  current_choice = (current_choice + 1) % len(route_decisions)
  # periodically re-shuffle
  if current_choice == 0:
    pass_count = (pass_count + 1) % shuffle_period
    if pass_count == 0:
      random.shuffle(route_decisions)
  return selected_route

The advantages of WRR are:

• Ensures all routes are used during one iteration of route_decisions.
• Prevents probabilistic imbalance, like that a route is excessively selected or vice versa.

As there are several adjustable parameters like multiplier and shuffle_period, I'd suggest to add configurations for load balancing in App Proxy.