Stuck at "reMarkable is starting" screen after uninstalling Koreader and performing a factory reset

[jhkkRM]

Issue:
Remarkable won't boot up after uninstalling Koreader and performed a factory reset.

Steps to reproduce
I just uninstalled Koreader, and enabled xochtil. It works well when I reboot the machine. But unfortunately after I performed the factory reset, the machine stops at "reMarkable is starting" screen.
I login to the machine, and execute "systemctl restart xochitl" with the following error message:
Warning: The unit file, source configuration file or drop-ins of xochitl.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Job for xochitl.service failed because the control process exited with error code.
See "systemctl status xochitl.service" and "journalctl -xe" for details.

Below is the details of
xochitl.service: Scheduled restart job, r
[[0;1;31mWarning:[[0m The unit file, source configuration file or drop-ins of xo
chitl.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[[0;1;31m●[[0m xochitl.service - reMarkable main application
Loaded: loaded (/lib/systemd/system/xochitl.service; enabled; vendor preset
: enabled)
Drop-In: /etc/systemd/system/xochitl.service.d
└─toltec-wrapper.conf
Active: [[0;1;31mfailed[[0m (Result: exit-code) since Thu 2022-02-10 23:56:
41 CST; 45min ago

Process: 242 ExecStart=/opt/bin/xochitl --system [[0;1;31m(code=exited, stat
us=203/EXEC)[[0m
Main PID: 242 (code=exited, status=203/EXEC)

Feb 10 23:56:41 reMarkable systemd[1]: xochitl.service: Scheduled restart job, r
estart counter is at 4.
Feb 10 23:56:41 reMarkable systemd[1]: Stopped reMarkable main application.
Feb 10 23:56:41 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxo
chitl.service: Start request repeated too quickly.[[0m
Feb 10 23:56:41 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxo
ssfully
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
Process: 242 ExecStart=/opt/bin/xochitl --system [[0;1;31m(code=exited, stat
us=203/EXEC)[[0m
Main PID: 242 (code=exited, status=203/EXEC)

Feb 10 23:56:41 reMarkable systemd[1]: xochitl.service: Scheduled restart job, r
estart counter is at 4.
Feb 10 23:56:41 reMarkable systemd[1]: Stopped reMarkable main application.
Feb 10 23:56:41 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxo
chitl.service: Start request repeated too quickly.[[0m
Feb 10 23:56:41 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxo
chitl.service: Failed with result 'exit-code'.[[0m
Feb 10 23:56:41 reMarkable systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to star
t reMarkable main application.[[0m
Feb 10 23:56:41 reMarkable systemd[1]: xochitl.service: Triggering OnFailure= de
pendencies.
Feb 10 23:58:26 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxo
chitl.service: Start request repeated too quickly.[[0m
Feb 10 23:58:26 reMarkable systemd[1]: [[0;1;38;5;185m[[0;1;39m[[0;1;38;5;185mxo
chitl.service: Failed with result 'exit-code'.[[0m
Feb 10 23:58:26 reMarkable systemd[1]: [[0;1;31m[[0;1;39m[[0;1;31mFailed to star
t reMarkable main application.[[0m

~

By the way, I tried to run "toltecctl uninstall" and get the feedback "toltecctl: command not found"
I tried to create the symlink again
Created symlink /etc/systemd/system/multi-user.target.wants/xochitl.service \342\206\222 /lib/systemd/system/xochitl.service
-sh: Created: command not found
Please help. Thanks.

[Eeems]

https://remarkable.guide/faqs.html#is-it-safe-to-factory-reset-my-device

https://toltec-dev.org/#frequently-asked-questions

[LinusCDE]

Hi,

while Eeems is very much correct here that we don't support factory resets and that the website warns about this, I'd still like to give it a shot to resolve the issue.

Luckily it seems you safe your SSH password beforehand and you can still log into it, which means it should be not too hard to get the device running again.

First, an easy way might be to manually execute "xochitl" and see if it launches. You can then check for a update of your device which should install a new, clean rootfs and that should work as usual again.

The problem with factory resetting the reMarkable without uninstalling toltec first is, that toltec stores the bulk of all it's data in /home/root/.entware. But it also needs to alter some parts of the remaining system in order to properly integrate with the device as well as having launchers work as expected. When factory resetting the reMarkable, it simply wipes /home/root afaik but leaves the remaining system untouched as it doesn't expecte the user to affect it in any way during normal usage. This causes the modified system files to point to non-existing files and thus have toltec fail to operate which prevents xochitl from starting up as well.

I'm not sure rn how the modification of the file xochitl.service is handled (the file used when running systemctl ... xochitl commands). It should be either in /etc/systemd/system/ or /lib/systemd/system/. It seems that the file has some external modification / override attached to it, but it also might have been updated to point to now missing toltec files completely.

There should also be a file opt.mount in one of those same directories which attempts to bind-mount /home/root/.entware as /opt during startup.

If the first idea didn't work, or there is no update available to you right now, you can try one of these two ideas as well. No guarantees they work or cause any harm, but might resolve it:

• If your previous update didn't have toltec installed you can roll your device back to it. The following file can do that. Save the content of this file to your remarkable (e.g. as /home/root/switch.sh, mark it as executable with chmod +x switch.sh and run it with ./switch.sh). After that a reboot should gave you boot your previous update with a fresh update to the newest / current release being downloaded pretty quickly which should fix stuff. (I think i found that script inside of ddvk's scripts at some point, but believe it's no longer there.)
• Otherwise you could try to reinstall toltec and then properly uninstall it again. If the installer complains that toltec is already installed, removing the file /lib/systemd/system/opt.mount (followed by a systemctl daemon-reload might do the trick. The uninstall with toltecctl uninstall.

If you happen to install toltec again, make sure your firmware is not unsupported and make sure to read the FAQ this time (both on the Website).

As already mentioned by Eeems we generally do not provide support for bricking in this way as it is known to not work. However I had some free time and felt a small explanation as well as some proposed ideas on fixing it might help. I do not think any of these should brick your device further, but be careful and read and carefully consider any error messages that might crop up. I have not tested these ways on a System that is in a similar state as yours, so I can't guarantee success or that it might cause further harm.

[jhkkRM]

Just find the files xochitl.service and opt.mount in folder /lib/systemd/system/. I just rename the file opt.mount, run systemctl daemon-reload, and then systemctl restart xochitl, but get the same error that
Warning: The unit file, source configuration file or drop-ins of xochitl.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Because the RM stuck at the startup screen, there is no network connection. In this case, I cannot install toltec.
Below is the content of opt.mount:

root@reMarkable:/lib/systemd/system# vi opt.mount.backup
[Unit]
Description=Bind mount /home/root/.entware over /opt
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target
[Mount]
What=/home/root/.entware
Where=/opt
Type=none
Options=bind
[Install]
WantedBy=local-fs.target

I tried to see the content of xochitl.service. It says that

root@reMarkable:/lib/systemd/system# vi xochitl.service
[Unit]
Desed: no

    process ID: 345

While opening file "xochitl.service"
dated: Wed Mar 30 21:45:36 2022
NEWER than swap file!

(1) Another program may be editing the same file. If this is the case,
be careful not to end up with two different instances of the same
file when making changes. Quit, or continue with caution.
(2) An edit session for this file crashed.
If this is the case, use ":recover" or "vim -r xochitl.service"
to recover the changes (see ":help recovery").
If you did this already, delete the swap file ".xochitl.service.swp"
to avoid this message.

Swap file ".xochitl.service.swp" already exists!
cription=reMarkable main application
StartLimitIntervalSec=600
StartLimitBurst=4
OnFailure=remarkable-fail.service
After=home.mount

Any further advice?

[Eeems]

Have you tried the commands in the FAQ section of the website I linked yet?

[jhkkRM]

After I input the following commands, it works now.
rm /etc/systemd/system/xochitl.service.d/toltec-wrapper.conf
systemctl daemon-reload
systemctl reset-failed xochitl
systemctl restart xochitl