Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Upgrade from 1.76.0 to 1.95.1 results in synapse.storage.background_updates: [background_updates-0] Error doing update: 'int' object is not iterable #16591

Closed
Artiom-M opened this issue Nov 1, 2023 · 3 comments · Fixed by #16594
Closed

Upgrade from 1.76.0 to 1.95.1 results in synapse.storage.background_updates: [background_updates-0] Error doing update: 'int' object is not iterable #16591

Artiom-M opened this issue Nov 1, 2023 · 3 comments · Fixed by #16594
Assignees
@clokep
Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release

Comments

@Artiom-M
Copy link

Artiom-M commented Nov 1, 2023 •
edited
Loading

Description

After standard upgrade with pip service starts and apparently working, but log is full of errors about
Error doing update: 'int' object is not iterable
matrix.log

Homeserver

self hosted

Synapse Version

{"server_version":"1.95.1"}

Installation Method

PIP

Database

PostgreSQL single server

Workers

Single process

Platform

Oracle linux 8

Configuration

No response

Relevant log output

Nov 01 05:06:58  matrix-synapse[3419535]: 2023-11-01 05:06:58,673 - synapse.storage.background_updates - 412 - ERROR - background_updates-0 - Error doing update: 'int' object is not iterable
Nov 01 05:06:58  matrix-synapse[3419535]: Traceback (most recent call last):
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/background_updates.py", line 409, in run_background_updates
Nov 01 05:06:58  matrix-synapse[3419535]:     result = await self.do_next_background_update(sleep)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/background_updates.py", line 540, in do_next_background_update
Nov 01 05:06:58  matrix-synapse[3419535]:     await self._do_background_update(desired_duration_ms)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/background_updates.py", line 583, in _do_background_update
Nov 01 05:06:58  matrix-synapse[3419535]:     items_updated = await update_handler(progress, batch_size)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/databases/main/pusher.py", line 635, in _set_device_id_for_pushers
Nov 01 05:06:58  matrix-synapse[3419535]:     nb_processed = await self.db_pool.runInteraction(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 934, in runInteraction
Nov 01 05:06:58  matrix-synapse[3419535]:     return await delay_cancellation(_runInteraction())
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
Nov 01 05:06:58  matrix-synapse[3419535]:     result = context.run(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
Nov 01 05:06:58  matrix-synapse[3419535]:     return g.throw(self.type, self.value, self.tb)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 900, in _runInteraction
Nov 01 05:06:58  matrix-synapse[3419535]:     result = await self.runWithConnection(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 1029, in runWithConnection
Nov 01 05:06:58  matrix-synapse[3419535]:     return await make_deferred_yieldable(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/threadpool.py", line 244, in inContext
Nov 01 05:06:58  matrix-synapse[3419535]:     result = inContext.theWork()  # type: ignore[attr-defined]
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
Nov 01 05:06:58  matrix-synapse[3419535]:     inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/context.py", line 117, in callWithContext
Nov 01 05:06:58  matrix-synapse[3419535]:     return self.currentContext().callWithContext(ctx, func, *args, **kw)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/python/context.py", line 82, in callWithContext
Nov 01 05:06:58  matrix-synapse[3419535]:     return func(*args, **kw)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib/python3.8/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
Nov 01 05:06:58  matrix-synapse[3419535]:     result = func(conn, *args, **kw)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 1022, in inner_func
Nov 01 05:06:58  matrix-synapse[3419535]:     return func(db_conn, *args, **kwargs)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 762, in new_transaction
Nov 01 05:06:58  matrix-synapse[3419535]:     r = func(cursor, *args, **kwargs)
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/databases/main/pusher.py", line 614, in set_device_id_for_pushers_txn
Nov 01 05:06:58  matrix-synapse[3419535]:     self.db_pool.simple_update_many_txn(
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 2078, in simple_update_many_txn
Nov 01 05:06:58  matrix-synapse[3419535]:     args = [tuple(x) + tuple(y) for x, y in zip(value_values, key_values)]
Nov 01 05:06:58  matrix-synapse[3419535]:   File "/home/matrix/synapse/env/lib64/python3.8/site-packages/synapse/storage/database.py", line 2078, in <listcomp>
Nov 01 05:06:58  matrix-synapse[3419535]:     args = [tuple(x) + tuple(y) for x, y in zip(value_values, key_values)]
Nov 01 05:06:58  matrix-synapse[3419535]: TypeError: 'int' object is not iterable

Anything else that would be useful to know?

No response
@erikjohnston
Copy link
Member

Hi @Artiom-M, thanks, this indeed looks like a bug. It's in a background update though so shouldn't affect day-to-day operation beyond spamming the logs with errors. Usually I'd suggest downgrading again, but since v1.95.1 is a security release you might want to stay on v1.95.1.

The problem is:

synapse/synapse/storage/databases/main/pusher.py

Lines 617 to 618 in a115119

key_names=("id",),
key_values=[row[0] for row in rows],

That second line should be a list of tuples, i.e. [(row[0],) ...]

Broke in #16431.

cc @clokep.

@erikjohnston erikjohnston added S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release O-Occasional Affects or can be seen by some users regularly or most users rarely labels Nov 1, 2023
@clokep clokep self-assigned this Nov 2, 2023
@clokep clokep mentioned this issue Nov 2, 2023
Merged
@clokep
Copy link
Member

clokep commented Nov 2, 2023

#16594 should fix this.

@Artiom-M
Copy link
Author

Artiom-M commented Nov 3, 2023 •
edited
Loading

As this is apparently not breaking anything for me, I will stay with 1.95.1 waiting for release.
Thank you.

P.S. Actually I have just applied the fix manually to the file.

@Artiom-M Artiom-M closed this as completed Nov 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@clokep clokep

Labels
O-Occasional Affects or can be seen by some users regularly or most users rarely S-Minor Blocks non-critical functionality, workarounds exist. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Regression Something broke which worked on a previous release
Projects
None yet
Milestone
No milestone
3 participants
@clokep @Artiom-M @erikjohnston