Problems with MFA under some load

[ghosftflc]

Description of the working infrastructure

1. 2 DNS records in cloudflare (round robin) for the service name
2. 2 haproxy in two data centers, to which the records from cloudflare refer
3. 2 backends on each haproxy. The backend is an adfs+mfa server.
4. Storage - MSSQL (a separate server in one of the data centers).

Problems are observed when the load on the service increases (approximately after 25-30 users per minute):

1. The service response time increases sharply (visible from haproxy statistics)
2. In the adfs-mfa server logs, errors like "Error on Check Remote Service method: adfs02.xxxxxxx.xxx => This request operation sent to net.tcp://adfs02.xxxxxxx.xxx:5987/ReplayService did not receive a reply within the configured timeout (00:00:15). ..."
3. Errors in responses to users:

• "The operation completed successfully", but there is no login to the target service.
• "Error during authentication process! One Time Password has been played before" when entering the correct code once.
• "Incorrect identification, please try again." when entering the correct code.

To find the cause of the problem, we prepared a test stand and conducted many load tests.
I will not provide the entire list of possible combinations, I will only describe the infrastructure options in which the service worked normally without any errors for users and visible problems:

Infrastructure option 1

1. 1 DNS record in cloudflare for the service name
2. 1 haproxy in one data center, referenced by the record from cloudflare
3. 1 backend on haproxy. The backend is an adfs+mfa server (there is one registered in the mfa cluster).
4. Storage - MSSQL (a separate server in the data center).

Infrastructure option 2

1. 1 DNS record in cloudflare for the service name
2. 1 haproxy in one data center, referenced by the record from cloudflare
3. 1 backend on haproxy. The backend is an adfs+mfa server, but there are two registered servers in the mfa cluster. The second one is enabled, but requests from haproxy do not go to it.
4. Storage - MSSQL (separate server in the data center).

Please help me understand the causes of the problem.
Perhaps this is the expected behavior of the service?
Please suggest proven recommended schemes with balancing of this mfa service.

[redhook62]

Hi, @ghosftflc

I can assure you that it is not with 20 or 30 simultaneous connections that performance issues appear.
First, as indicated in the documentation

• TCP port 5987 must be open on each ADFS server (use Set-MFAFirewallRules on each server)
• The fqdn of each server must respond in the DNS (using entries in the host files is acceptable and sometimes recommended)
• Routing between your ADFS servers must be fast...
  You can disable Anti-Replay to disable caller synchronization between your different servers.
  and then restart the MFA service.

regards

[ghosftflc]

Thanks for the recommendations. Please remind me how I can disable Anti-Replay?

[redhook62]

In the Snappin MMC

https://github.com/neos-sdi/adfsmfa/wiki/07-Security#managing-mfa-security-with-mmc

[ghosftflc]

Do I need the "Replay protection level" to be set to "Disable"?

[redhook62]

Yes, Of course

regards

[ghosftflc]

I set the Replay Protection Level parameter to Disabled and conducted several tests, but the result did not change. The exchange between ADFS/MFA nodes is in progress, and there are still many errors in the log, which I wrote about in the request. What else can I try to configure to solve the problem?

[redhook62]

did you restart the MFA service as instructed?
PS> restart-service mfanotifhub

[ghosftflc]

Yes. I did. I restarted the service many times after the parameter was disabled.

[redhook62]

The only thing I can advise you is to check your environment.
Have all the prerequisites been followed ? What is your configuration ? etc...
Otherwise, switch to another solution

regards

[ghosftflc]

I described my configuration in the initial request. Similar configurations are described in your community and judging by the reviews they are quite working.

All the prerequisites have been followed:
TCP port 5987 is open on each ADFS server
"The fqdn of each server must respond in the DNS" - We use haproxy on the front, and ADFS servers are backends in haproxy.
Routing between our ADFS servers is fast

[redhook62]

From experience I can't take your words for granted. I have clients who have tens of thousands of users, and everything works fine.
So do as you feel, if you think it's the component's fault then why continue, move on.
Otherwise at least as I asked please export your configuration and post it to us

[ghosftflc]

I have uploaded the configuration file. Please take a look at it.
config.txt

[redhook62]

I doubt very much that on your test Lab you can saturate the entry of TOTP codes.
First of all, it is in no way a production platform with a representative number of users.
Your test platform is probably undersized.
Why are you using SQL Server Encrypted Columns? TOTP keys are encrypted.
Why not superuser for the connection.
With your test domain names, check all DNS entries.
The problem is definitely on your side, your lab is not "optimized".
Look on your side, and stop blaming the component...

regards