User Directory search does not return user from public room with limit parameter

[weeman1337]

Description

/_matrix/client/v3/user_directory/search with a limit does not return users that are in a public room and I don't share a room with

Steps to reproduce

• Create some users
• Let one of the users join a public room
• Query a user from the public room with another user, that does not share a room with the other users
• Be sure the in addition to the query term there is also a limit of 50 set

Expected:
Other user with no shared room, but in a public room should be returned

spec „the homeserver MUST at a minimum consider the users the requesting user shares a room with and those who reside in public rooms“

What happens:
Directory search does not return the user

Homeserver

localhost

Synapse Version

d6b7d49

Installation Method

Docker (matrixdotorg/synapse)

Database

SQLite

Workers

Single process

Platform

• Podman locally
• Docker for Element Web CI

Configuration

No response

Relevant log output

2023-10-12 11:22:10,546 - synapse.access.http.8008 - 465 - INFO - POST-159 - ::ffff:10.0.2.100 - 8008 - {@user_3:localhost} Processed request: 0.001sec/0.000sec (0.000sec, 0.000sec) (0.000sec/0.001sec/1) 30B 200 "POST /_matrix/client/v3/user_directory/search HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.17.3 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36" [0 dbevts]
2023-10-12 11:22:11,293 - synapse.access.http.8008 - 420 - DEBUG - POST-160 - ::ffff:10.0.2.100 - 8008 - Received request: POST /_matrix/client/v3/user_directory/search
2023-10-12 11:22:11,294 - synapse.storage.txn - 742 - DEBUG - POST-160 - [TXN START] {search_user_dir-59a}
2023-10-12 11:22:11,294 - synapse.storage.SQL - 453 - DEBUG - POST-160 - [SQL] {search_user_dir-59a} SELECT d.user_id AS user_id, display_name, avatar_url FROM user_directory_search as t INNER JOIN user_directory AS d USING (user_id) LEFT JOIN users AS u ON t.user_id = u.name WHERE ( EXISTS (select 1 from users_in_public_rooms WHERE user_id = t.user_id) OR EXISTS ( SELECT 1 FROM users_who_share_private_rooms WHERE user_id = ? AND other_user_id = t.user_id ) ) AND (u.locked IS NULL OR u.locked = FALSE) AND value MATCH ? ORDER BY rank(matchinfo(user_directory_search)) DESC, display_name IS NULL, avatar_url IS NULL LIMIT ?
2023-10-12 11:22:11,294 - synapse.storage.SQL - 458 - DEBUG - POST-160 - [SQL values] {search_user_dir-59a} ('@user_3:localhost', '(bytebot* OR bytebot)', 51)
2023-10-12 11:22:11,294 - synapse.storage.SQL - 479 - DEBUG - POST-160 - [SQL time] {search_user_dir-59a} 0.000075 sec
2023-10-12 11:22:11,294 - synapse.storage.txn - 846 - DEBUG - POST-160 - [TXN END] {search_user_dir-59a} 0.000268 sec
2023-10-12 11:22:11,295 - synapse.access.http.8008 - 465 - INFO - POST-160 - ::ffff:10.0.2.100 - 8008 - {@user_3:localhost} Processed request: 0.001sec/0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/1) 30B 200 "POST /_matrix/client/v3/user_directory/search HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Cypress/12.17.3 Chrome/106.0.5249.51 Electron/21.0.0 Safari/537.36" [0 dbevts]

Anything else that would be useful to know?

Found with one of the Element Web end-to-end tests flaking. With the test this occurs quite often.

[reivilibre]

Please could you get us some information from the database?

SELECT * FROM users_in_public_rooms;


SELECT * FROM user_directory_stream_pos;

SELECT * FROM user_directory;

How long are you waiting before searching for the user? The user is added in the background so there might be a small delay (or a big one on an overloaded homeserver, I guess).

[weeman1337]

How long are you waiting before searching for the user? The user is added in the background so there might be a small delay (or a big one on an overloaded homeserver, I guess).

Maybe I didn't wait long enough. If I keep the server running for another minute or so the user appears. Closing the issue then.

[weeman1337]

Reopening. It seems that with a limit for the directory query it does not return the user. It does return the profile without a limit.

Query results:

Users in public rooms:

user_id          |room_id                      |
-----------------+-----------------------------+
@bot_5:localhost |!vLHsytDwrygpQQlCqT:localhost|
@user_3:localhost|!vLHsytDwrygpQQlCqT:localhost|
@bot_5:localhost |!STcsznzbbjeHFwckzs:localhost|
@bot_7:localhost |!STcsznzbbjeHFwckzs:localhost|
@bot_5:localhost |!NhsXZfwzrpIPUjZFag:localhost|
@bot_7:localhost |!NhsXZfwzrpIPUjZFag:localhost|

User directory stream pos:

Lock|stream_id|
----+---------+
X   |       24|

User directory:

user_id          |room_id|display_name|avatar_url|
-----------------+-------+------------+----------+
@bot_5:localhost |       |BotBob      |          |
@bot_7:localhost |       |ByteBot     |          |
@user_3:localhost|       |Jim         |          |

[DMRobertson]

What happens: Directory search does not return the user

@weeman1337 can you spell out what is happening in this animation? It looks like

• ByteBot makes a public room and invites BotBob
• BotBob accepts
• Some third user searches for ByteBot

I am just wondering if there is something more going on because EW shows the public room under the historical section?

[DMRobertson]

Alternatively... is there any way I can run this test locally to investigate?

[DMRobertson]

Tried to reproduce this as follows, but passes for me:

From b348c52c519f2e74ab77a6ae1bf8a6b0e0c05e05 Mon Sep 17 00:00:00 2001
From: David Robertson <[email protected]>
Date: Fri, 27 Oct 2023 17:47:47 +0100
Subject: [PATCH] Test that users in public rooms are searchable

---
 tests/rest/client/test_user_directory.py | 65 ++++++++++++++++++++++++
 1 file changed, 65 insertions(+)
 create mode 100644 tests/rest/client/test_user_directory.py

diff --git a/tests/rest/client/test_user_directory.py b/tests/rest/client/test_user_directory.py
new file mode 100644
index 0000000000..02d4ff5e35
--- /dev/null
+++ b/tests/rest/client/test_user_directory.py
@@ -0,0 +1,65 @@
+# Copyright 2023 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from http import HTTPStatus
+
+from synapse.rest import admin
+from synapse.rest.client import login, room, user_directory, profile
+
+from tests import unittest
+from tests.unittest import override_config
+
+
+class UserSearchTestCase(unittest.HomeserverTestCase):
+
+    servlets = [
+        admin.register_servlets_for_client_rest_resource,
+        login.register_servlets,
+        room.register_servlets,
+        user_directory.register_servlets,
+        profile.register_servlets,
+    ]
+
+    @override_config({
+        "update_user_directory_from_worker": None,
+    })
+    def test_user_directory_search_returns_users_in_public_rooms(self) -> None:
+        alice = self.register_user("alice", "pass")
+        alice_token = self.login(alice, "pass")
+        bob = self.register_user("bob", "pass")
+        bob_token = self.login(bob, "pass")
+
+        # Alice sets a displayname.
+        channel = self.make_request(
+            "PUT",
+            f"/profile/{alice}/displayname",
+            content={"displayname": "Alice"},
+            access_token=alice_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.json_body)
+
+        # Alice creates a public room.
+        self.helper.create_room_as(alice, True, tok=alice_token)
+
+        # Bob searches for Alice by her displayname.
+        channel = self.make_request(
+            "POST",
+            "/_matrix/client/v3/user_directory/search",
+            {
+                "search_term": "Alice",
+            },
+            access_token=bob_token,
+        )
+        self.assertEqual(channel.code, HTTPStatus.OK, channel.json_body)
+        self.assertEqual(channel.json_body.get("results")[0]["user_id"], alice, channel.json_body)
+
-- 
2.41.0