Full Backup (i.e. via D2D) kills apps, including always-on VPNs, and they do not typically get restarted

[t-m-w]

Perhaps unexpectedly, AOSP sends a kill signal to an application's main process after a full backup has been completed rather than before. But either way, the unfortunate result is that apps which run with a single process may remain affected until the user intervenes. This includes apps that are expected to stay running, like always-on VPN apps. A user's device could lose connectivity entirely, which is particularly problematic if the backup happens while the device is unattended, since the user would not know anything was wrong until they looked at their device. (Also of note, in the case of a "lockdown" VPN, if Seedvault is performing a network backup, this will likely cause the backup to fail.)

A workaround is to exclude affected apps from the backup, but this is not ideal.

Here are some links I gathered while looking into this issue that may be informative:

• Killing of the application process, today
• Original justification (when just for restore)
• First commit for full backup infrastructure
• Default full backup mechanism
• Binding of the backup agent

And here's my original comment regarding this issue, before I'd started researching the cause:

I've noticed some issues lately, likely related to D2D backups, that are going to be frustrating for users. For example, some VPNs are not getting automatically restarted anymore after they are backed up, even when set as Always-On VPNs. I don't think this is specific to app-backup-v2, but since D2D is the default and only option now, we should be aware of it. Likely I will want to copy/paste some of this comment into another issue or two somewhere later.

Known-affected VPNs:

• Mullvad 2024.4 from F-Droid. (2024.3 is fine.) It needs to be manually started after a backup, from that version forward.
• WG Tunnel 3.5.2 from F-Droid. It needs to be manually started after a backup.
• Orbot 17.3.2-RC-1-tor-0.4.8.12. It needs to be manually started after a backup.
• Wireguard 1.0.20231018. It needs to be manually opened after a backup.

Some other apps seeing unwanted effects:

• Auxio 3.5.3 from F-Droid. It loses its current playing track and position and shuffle state (probably just everything, killed without saving it; force-stop does the same thing).

I'll keep an eye out for others.

The VPN issues are especially notable, since for some configurations, the device might lose network connectivity in the middle of the night or day. If the backup takes place over the VPN, that could mess up that backup. The simplest suggestion for now is of course to exclude the VPN app from backups entirely.

On second thought, as far as VPNs, this could be an AOSP bug, since always-on VPNs should get restarted regardless in my opinion.

Anyway, my hope is that issues like this can often be addressed in the apps themselves, especially in the case of Mullvad for example, where we have a known-working and known-broken version for comparison.

[grote]

Interesting that backupInForeground doesn't seem to get respected. It is false by default.

Maybe this is only when the system requests the backup itself and not when we request it with own scheduling?

[grote]

Also relevant here: android:killAfterRestore="true"

[eternal-sorrow]

Please fix it. Because there is no convenient way to set the time for backup, this is very frustrating. It started with the latest LineageOS update.

[t-m-w]

Please fix it. Because there is no convenient way to set the time for backup, this is very frustrating. It started with the latest LineageOS update.

Yes, it is frustrating. Are you experiencing this with a VPN app, too, or something else? While it's not ideal, like I mentioned in the main post, you can exclude the VPN app for now, which is what I've taken to doing.

[eternal-sorrow]

@grote No, it's not VPN for me. But thanks for the advice, I'll try excluding apps.

[t-m-w]

@eternal-sorrow Okay, cool. And you don't have to share which apps / types of apps are affected for you, but I think it could be useful to know and to help put this issue into context. Right now, mainly I know always-on VPN apps are a big one.

[eternal-sorrow]

Well, I have a learning routine when I use a learning app every day at the same hour. And it coincided with the time when Seedvault did the backup. The app was killed right while I was using it and I lost the progress of that learning session. I sorted it out by temporarily disabling the backup schedule and then reenabling it at a time when I won't be using the learning app. I had to do it this way because Seedvault has no way to control the time when the backup will happen, only frequency.

The other app that I have problems with is NewPipe. I use a feature of queue in NewPipe quite extensively, queueing the videos that I want to watch later. But NewPipe only persists the queue while it's running in the background. When the app is killed and restarted, the queue is lost. I can save the queue contents to a playlist if I want to, but it's a chore, so I'll just exclude NewPipe from the backup.

[t-m-w]

@eternal-sorrow Thanks, that's useful information and good further justification for #780 I think.

Note that it's not Seedvault which is killing the apps, but the system. Seedvault itself can't directly prevent that, as far as I know so far.

If we were limited to addressing this within Seedvault itself, at best we may be able to re-start apps (which would not address your loss of progress, etc) or allow not using D2D for backup again, to work around this. The former is not ideal, if it's even possible, and neither is the latter, since it means less of your data gets backed up and Seedvault having to support both scenarios. Anything else would have to be handled in the system (which can be done, but requires system modifications). So this makes #780 all the more appealing.

But take some of this with a grain of salt because it's still being researched.

[eternal-sorrow]

D2D is Device-to-device backups, which is enabled in expert settings, correct? That option allows to backup most of the apps, even those that don't allow backup normally. If that's the case, than a perfect solution for me would be to be able to select which apps will use D2D and which won't. The apps that won't use D2D will use normal backup if possible. And also an info box notifying me that D2D kills running apps would be nice I guess.

[grote]

If that's the case, than a perfect solution for me would be to be able to select which apps will use D2D and which won't.

@t-m-w Is it d2d type of backup killing apps or backup in general?

Even if it is d2d causing the killing, we can't really do d2d only for some apps and not for others. When doing the backup, we have to tell the system what kind of backup we do and what kind of backup transport we are.

If you want exclude apps, then use the exclude apps from backup feature in backup status.

[t-m-w]

@t-m-w Is it d2d type of backup killing apps or backup in general?

@grote Not entirely sure, sorry. Needs more research.

[eternal-sorrow]

Even if it is d2d causing the killing, we can't really do d2d only for some apps and not for others

If you want exclude apps, then use the exclude apps from backup feature in backup status.

Don't you see a contradiction here? If you can exclude apps, then you CAN do d2d only for some apps and not for others.

[chirayudesai]

Don't you see a contradiction here? If you can exclude apps, then you CAN do d2d only for some apps and not for others.

No, because D2D applies to the entire backup. It makes things better for most apps, but for some it's causing issues, as noted here.

The options:

1. Don't do D2D only for the apps having issues - not possible, the entire backup has to be D2D, or not. Can't say make one app D2D, but not this other app. It's all or nothing.
2. Exclude the apps having issues from backup entirely - only current solution, but that means there's no backup at all of the affected apps.
3. Don't do D2D at all - will fix these apps, but make backup worse for other apps. Not toggle-able anymore.
4. Something else, if we can figure that out.

[eternal-sorrow]

@chirayudesai

5. Do D2D backup for most apps excluding some and then normal backup for those excluded apps.

[grote]

Do D2D backup for most apps excluding some and then normal backup for those excluded apps.

Backup is snapshot based. . You will end up with two snapshots each containing a disjoint set of apps. There may be some hackery we could attempt to unify both, but I am not willing to entertain such brittle hacks when noone has even done the research needed to understand the cause of the issue, yet.

[eternal-sorrow]

Has anyone tried to ask AOSP developers? Because this issue started not long ago, with the latest update of Lineage OS for me (25 October), so it was probably regressed by some changes in AOSP.

[grote]

You probably only noticed it now, because we now do d2d which includes more apps into the backup. Before, few apps got backed up, so the ones you use daily probably weren't affected.

[eternal-sorrow]

I have D2D enabled for quite some tome now (at least from August this year).

[t-m-w]

@eternal-sorrow Thanks again for all this info. It'll be helpful as we dig into this further.

[t-m-w]

Per @grote's recommendation, I installed an older version of CalyxOS to check the behavior there - I chose 5.4.0 from February 2024. With the D2D toggle turned on, Mullvad does indeed get killed and does not get restarted. I was using the latest version of Mullvad, which within the last few months was changed to only use a single process. Previously the VPN connection was in a different process that did not get killed.

If I turn off D2D backup and run it again, Mullvad does not get killed, but Mullvad does not allow backup in this case, so it's not even attempted.

I have been speculating, based on some behavior I noticed on my daily driver, that app stores' app update notifications may be affected by this process-killing, but I have not confirmed that yet.

[t-m-w]

Tested again with Rethink (still February's CalyxOS 5.4.0), which does not prevent backup, so I was able to turn off D2D, and I did so. Running a backup does kill Rethink, and it does not get restarted - so, with always-on + block connections without VPN, there is no network access at all until Rethink is manually launched.

That means this problem, at least as it pertains to always-on VPNs, has existed for a while. But it would still be interesting to try an even earlier OS and see what happens.

Edit:
I went back as far as Android 13 via CalyxOS 4.14 from October 2023. This was before D2D was even an option. Rethink gets killed there too, and not restarted, so as far as always-on VPNs go, this is a long-standing problem. D2D merely exacerbates it. Many VPN apps - e.g. Calyx VPN, Mullvad, WG Tunnel - simply disallow backup. This makes them typically unaffected.

Edit 2:
As @grote has pointed out before, the system is supposed to "[shut] down the app to make sure it is no longer writing to the file system", but this is not the behavior I have observed lately, nor was I able to find where this should be happening in code via Android Code Search. Maybe someone else can. But if my observation is correct that an app and/or its main process do not get killed before backup, and only after, this seems like it could lead to weird results or corruption for the very reason AOSP gives about filesystem writes, and it is somewhat surprising that this issue hasn't been noticed yet.

I uploaded an experimental change to the CalyxOS Gerrit which performs backup in a separate process. So far, based on limited real-world use in which only two backups have taken place, it looks like it is working okay. I have observed that neither Mullvad nor my music player app gets killed anymore - it is still running even without being specifically excluded - and the music player does get successfully backed up, which I know because a restore brought it to the expected state. Even with the prior single-process approach, I don't see what would stop an app from modifying its own files in a separate thread while a backup is taking place, so in my mind, using a separate process primarily just makes it easier to kill that process (and only that process) later.

On the other hand, if an app were truly killed before backing up (i.e. if this apparent upstream bug were fixed, or if I have been somehow misinterpreting everything), then the above change wouldn't be applicable anymore, but we / AOSP / someone may still need to consider how to restart it.

[t-m-w]

Reported to AOSP: https://issuetracker.google.com/issues/381899617

...and emailed a couple of the BACKUP_OWNERS developers.