Slow POST /auth/token

[smadbe]

Calls to /auth/token?cookie_same_site=1&cookie_secure=1&create_temp_user_if_not_authorized=1&default_language=fr&use_cookie=1 seems to be (very) slow from time to time while it looks like a simple service. This service is very important as it is called for every not-already-connected user arriving on the website.

After, after the v2.23.0 release, I still see on this service "unexpected error: lock wait timeout exceeded" errors and services taking >3sec (maybe corresponding to this timeout).

Note: POST /auth/token?cookie_same_site=:id&cookie_secure=:id&use_cookie=:id is also slow with a 728ms 95% percentile in the 3 last hours... so maybe it is not necessarily related with the "create_temp_user_if_not_authorized".

[zenovich]

The issue consists of several things:

1. We do not retry (and never retried) on timeouts on named locks acquiring.
2. The timeout for named locks is only 3 seconds (can be fixed by Raise groupsRelationsLockTimeout & itemsRelationsLockTimeout to 50 seconds #1198)
3. The relations (items_items/groups_groups) creation/deletion is always done under named locks, one named lock for all the operations modifying each graph. This means only one session can modify the groups relation graph at each moment (e.g. we can register only one temporary user at a time, and we cannot register users concurrently with other groups graph modifications like adding/removing group relations). The same is true for the items relations graph.

I'm going to check if we can get rid of the named locks for items/groups relations

[zenovich]

We moved groups&items ancestors propagation back into initiating transactions in #1195.
Then we got rid of the named lock used for groups ancestors propagation and sped up the propagation by 30% in #1200.
Then we made the groups ancestors propagation twice faster in #1210.

If the service is still slow, we can try getting rid of the propagation in it completely.

[smadbe]

After these changes, stats of this services (over 1800 calls): avg: 214.4243, pct 95: 837.6792, max 7345.4092.
That's is not extremely slow but still not fast (especially for the one who got the >7s)