OBP Performance Tests

Scope

This document aims to answer the question: “How do we serve X thousand concurrent Users using OBP?”

Thus we first need to define: “What do we mean by a concurrent user?”

Let's define a User as: “A user using a mobile App for 5 minutes and making 6 API calls.”

An average user session lasts 5 minutes.

|-----|

In an hour we will have 60 / 5 = 12 of these 5 min sessions:

|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|

So our famous “Concurrent User” will make 72 calls in an hour.

So if we want to support 50,000 concurrent users: We will handle 50,000 * 72 = 3,600,000 requests per hour

i.e. 50,000 concurrent users... Make 1000 requests per second.

We want to support an average response time of under 500ms.

So the question is: “How many OBP instances do we need to support this?”

Method

To measure performance of an OBP API system, we use a JMeter test plan that simulates a typical client facing Application making a range of HTTP requests (calls) to the API.

This Test Plan reflects a simple AISP (Transaction history) and PISP (Payment) scenario.

Although authentication is not typically part of the main OBP-API responsibility we include a simple authentication call for ease of running the tests. Also, the DirectLogin API call is by far the most expensive in the tests.

1. DirectLogin: Login with username and password, generate a login token.
2. GetCurrentUser: Gets details about the currently logged in user
3. GetAccountsAtBank: List all bank accounts the user has
4. GetBalances: Gets balances of all bank accounts of the user
5. GetTransactions: Gets transaction history of one bank account
6. CreateTransactionRequest: Initiates a bank transfer

In between these API calls, there are pauses of random length.

The OBP instance tested consisted of a single-host installation with all components on one virtual machine: OBP API, database backend (PostgreSQL), and reverse http proxy (NGINX). The VM had 4 dedicated vCPUs and 8GB of RAM.

The test plan was configured to run over 5 minutes during which a total number of 1000 simultaneous iterations of the simulated app were run from a separate JMeter VM over a 1000MBit/s network connection. Each iteration was using a different user account on the instance. This test plan was executed several times, with similar results every time.

Results

The response times during one execution averaged at about 150ms per API call; without DirectLogin, the average is around 50ms per call. Detailed metrics are as follows:

See a recent JMeter test session here (JMeter files of the test plan here)

A near linear scaling of this setup can be assumed, as all the components can be run on separate and dedicated VMs or containers. I.e. If the load balancers, database cluster and OBP API instances all run with dedicated resources, they will be able to perform their dedicated tasks.

The results of our load test show that one OBP instance can serve 1000 concurrent users. Therefore, 10 instances will be required for 10,000 concurrent users and 50 instances for 50,000 concurrent users.