Provide cron and precompiled rust program to clean unreferenced states

[Gredin67]

Problem

• Possibly solve synapse DB huge size issues caused by unreferenced state groups #478

Solution

• provide precompiled
• run it in a cron if a mautrix bridge is installed
  • by checking installed apps named mautrix
  • or by checking presence of registration file named mautrix

PR Status

• Code finished and ready to be reviewed/tested
• The fix/enhancement were manually tested (if applicable)

Automatic tests

Automatic tests can be triggered on https://ci-apps-dev.yunohost.org/ after creating the PR, by commenting "!testme", "!gogogadgetoci" or "By the power of systemd, I invoke The Great App CI to test this Pull Request!". (N.B. : for this to work you need to be a member of the Yunohost-Apps organization)

[Gredin67]

rather than a cron, the cleanup could be run in the backup script, to avoid backing up a huge amount of useless data. (last time I run this cleanup I shrinked my synapse DB from 85G down to 16G :D

[Josue-T]

Thank you for your contribution,

I would really prefer to have the source and/or the precompiled binary somewhere else to avoid to put binary file in this repository which make it the repos bigger.

We can also completely discuss about putting the binary here https://github.com/YunoHost-Apps/synapse_python_build/releases

And also in case you provide the prebuild file (which is a great to avoid to build locally) you need to keep in mind to support:

• all debian version that this package support, so currently debian 11 and 12 and nearly debian 13
• all architecture that this package support so ideally: amd64, arm64, i386 and armv7

From what I understand this is doing a cleanup of element in the DB which is not used any more. If it's the case, I think we should integrate in the synapse package the cron maybe 1 time per week.

[Gredin67]

that would be perfect for me.
How can I provide a prebuilt if I don't have access to hardware of these architectures ? How do you proceed for synapse ?
The good thing is that it does not have to be maintained, as the clean up script has been stable for years. https://github.com/erikjohnston/synapse-find-unreferenced-state-groups
The only thing that can occur is that synapse solves the related bug :D

[Josue-T]

On this repos we have a script to build the binary we can just create an other script for this and since the script work well I will handle the build on the 4 architecture.

[Josue-T]

Note from the project:

CAUTION: While synapse is running and processing events there is a delay between the state groups and the events being persisted. As such this tool may pick up recent state groups that will in future get referenced by events that are still getting processed.

It seem that this tool need some analysis before running it. How do you think to handle this question ? Did you have some experience with this ?

[Gredin67]

I used it several times with no issue. The only precaution to take is to stop synapse while looking for unreferenced states, see my script in the PR. It is explained here erikjohnston/synapse-find-unreferenced-state-groups#8 Le 28 janvier 2025 08:43:46 GMT+01:00, Josue-T ***@***.***> a écrit :

…

Note from the project: > CAUTION: While synapse is running and processing events there is a delay between the state groups and the events being persisted. As such this tool may pick up recent state groups that will in future get referenced by events that are still getting processed. It seem that this tool need some analysis before running it. How do you think to handle this question ? Did you have some experience with this ? -- Reply to this email directly or view it on GitHub: #510 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

[Josue-T]

I used it several times with no issue. The only precaution to take is to stop synapse while looking for unreferenced states, see my script in the PR. It is explained here erikjohnston/synapse-find-unreferenced-state-groups#8

Ok, thanks I'll trust you on this side, so can you make a PR on this repos with the script to build the required binary ?

[Gredin67]

see YunoHost-Apps/synapse_python_build#1
It should probably be a two-liner script