Feature request: Reverse direction - encode backups

[nettnikl]

Hi!

What

I'd like to ask, if the reverse direction could also be implement without too much work.
Right now, i haven't yet deep dived into the code, but i assume there's no asymmetric part that would require any code from the original Signal devs, right? So given the needed data (clean text backup), you could encrypt it using this tool?
In case you need help testing or finding more information, maybe you could give me hints how to help?

Why

I'm trying to enable the use case described here: johanw666/Signal-Android#50 - given a backup of a Signal Desktop Instance (which can be decrypted by others tools like https://github.com/tbvdm/sigtop) i'd like to import this data back into Signal.

Issues

I assume that not all data is present in the Signal Desktop client that is available in a Signal Android client? But seeing that until recently-ish, Signal was able to import SMS (and ports with relatively lightweight changes can do so even now (and even WA backups): https://github.com/johanw666/Signal-Android) i think these fields can be left empty as optional.

[pajowu]

Hey, this sounds like a great idea, however I currently don't have enough time to implement it. This would also lead to us needing to update signal-backup-decode more often, so I would like to have someone who commits to maintain it, at least for a bit. Let me explain why:

Signal Backups consists of BackupFrames. Those frames can be attachments, stickers, prepared sql-statements and many more. The sql statements contain all the messages, groups etc. At the moment the code makes very little assumptions about the content of those statements. It just takes them (filters out a few "bad ones" that would lead to issues) and runs them against a sqlite database

Analysing the content of this database is left as an exercise to the reader (there are multiple tools to do that afaik). This way, signal-backup-decode does not need to be updated if signal adds a new table, changes the database layout etc. Only if they change the frame format, which they did for example to add stickers.

Those changes do not happen often, so it's not too much effort to do this for me. However to generate a "valid" signal backup we need to update the internal database format more often to match that of the real app.

While the "output" should be more stable, as the signal app migrates older versions on start. However the formats we get the data from (for example signal desktop backups) needs to be updated more often. I currently don't have the resources to do so, but would be happy to have someone else maintain it

[nettnikl]

Wow, thank you very very much for your valuable insights!

I can definitely provide some support, especially with smaller fixes, but cannot guarantee that I'll be able to keep it always up-to-date within the first days (unless there is a plan to introduce a ci pipeline somehow able to detect incompatibilities with the newest Signal version as soon as it is updated). If you tag me on issues (or i see them), I'll do my best to provide help.

About the technical details, I'll look into them and see what i can do. As mentioned there also is a project doing exactly the export of Signal Desktop backup, and I'll see if it seems reasonable to make use of this to get a more stable input, or atleast look through the commits to understand how often there are breaking changes in the Desktop backups.
Again, thank you very much!

[nettnikl]

Hey, so it org.thoughtcrime.securesms.database.MmsDatabase in Signal-Android contains the media data, but i'm having trouble connecting this to the format of

signal-backup-decode/proto/Backups.proto

Line 72 in 31100a3

message BackupFrame {

. Is the data contained there really everything that is needed in the backup database?

[pajowu]

The actual MmsDatabase-Class is less interesting for the backup itself. The whole export happens in the full backup exporter, which

1. write some metadata to the backup

2. copies the data from the databases (including mmsdatabase) to the backup file.

   This happens using the exportTable function, which basically generates a prepared sql statement for every row in the table, serializes the values of the row and then stores this in the backup file

   For some tables, the exporter also defines a post process function, which gets called after the row is added to the backup and copies some more data to the backup. One example is the export of the attachment table, where the post processing function writes the file itself into the backup.

   As you can see, this process is not specific to a single table / collection of tables. Instead it "blindly" serializes a table and the importer has to care about its content when importing it.

3. Copies the share prefs to the backup

4. backups some key-value store

All these backup functions use the BackupProtos classes, which are automatically generated from Backups.proto, which should be the same file as signal-backup-decode/proto/Backups.proto

I hope this answers your question, if not please feel free to ask again