-
Notifications
You must be signed in to change notification settings - Fork 118
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RichDocuments built-in CODE causes significant slowdown/timeout issues #1282
Comments
Hello, my Nextcloud instance is experiencing the same issue after version upgrade, although in my case it is enough to disable the Collabora Online - Built-in CODE Server app to solve the performance problem. Environment: Thank you for your time and effort in looking into this matter. Should you need additional information please let me know. |
Confirmed, this makes the snap unusable. Nothing interesting in the logs during usage that doesn't relate to collabora. When trying to open a document I see this though. Not sure if it's helpful:
Note that I've been trying to open that document for about 5 minutes now, it just keeps spinning. |
Got a lot of these in my PHP error log:
And this type of thing in PHP-FPM error log:
|
I found something on the nextcloud forum: hpilkington gave the tip: Confirmed sudo snap run nextcloud.occ app:disable richdocuments && sudo snap run nextcloud.occ app:disable richdocumentscore resolved my issues. After having done that, the web interface turns fast again. I hope there will be a fix soon for collabora... |
@juliushaertl I noticed similar issue probably introduced by: 6b3d3f1 |
Looked into the snap case with @timar. The findings so far (spoiler: the interesting part is at the end):
So far so good.
These aren't ideal, and might be worth investigating some other time (service may fail to kill the old version, and start the new one after an update), but isn't relevant now.
This is expected, there's no libfuse2 in snap, content of the AppImage is extracted to
loolwsd has started.
^ This indicates the problem. Only happens with version 6.4.202, while the previous 6.4.14 version is okay. |
@thebearon: I'd suggest removing the capabilities from the binaries when packaging this, that should make it working again. |
@juliushaertl: So we have investigated this more, and actually the problem is that the AppImage is run as root - who of course by definition has all the capabilities, so we refuse to start... Is it really necessary to run us as root? |
Actually, probably @kyrofa can answer this question. |
The loolwsd command should be ran as the same user that the webserver is using since richdocuments code is just calling PHP exec(), so I assume that this is rather snap specific. |
Yes, as we've discussed before, daemons in snaps run as root, although note that root in snaps is still confined so there are no special capabilities granted. |
Important to note that this issue isn't limited to snap, but has been reported by Docker image users also (though perhaps only those that installed everything as root?). Also note that one user on the Community Forum has said that upgrading to the edge release fixes the issue: https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304/27. It's just one user, of course. |
I use Nextcloud, latest version & fresh install, in Docker with the |
This also happens if nextcloud is installed on bare metal. It's not dependent on docker or snap. I have a collabora container on another VM, but nextcloud hasn't connected to it yet. Unfortunately in my case with nextcloud 20.04 the entire install errors out with richdocuments enabled so I can't configure it. With 20.02 I could at least load into the app after 45-60 seconds and once configured the issue went away. That's not possible on 20.04. |
I really hope a fix for this is forthcoming soon. I definitely is a big showstopper for me. |
richdocumentcode 6.4.203 has been released. It fixes the issue, i.e. now we let loolforkit run as root. |
I'm afraid I cannot confirm the fix. I upgraded to 6.4.203 and re-enabled the CODE server and Collabora Online and it has, unfortunately, knocked out my Nextcloud server (latest snap version) again. Disabling them brings performance right back. |
I can confirm, it still not working, logs:
nextcloud 20.0.4 (docker) |
The fix seems to work on snap. Although everything appears to be working a new notification appears (maybe unrelated?): "Invalid private key for encryption app. Please update your private key password in your personal settings to recover access to your encrypted files." Also many new errors are logged, mainly: [encryption] Warning: OC\ServerNotAvailableException: Legacy cipher is no longer supported! at <>
|
I tested this update this morning, using Docker, and it still acts the same : installing richdocuments & richdocumentscode result is veeerry slow instance, and it goes back to normal after uninstalling these apps. |
Can I ask, which version of snap are you on. I'm on the latest stable version and re-enabling the CODE server and Collabora Online after updating the app had no positive impact at all, but just brought back the same problem. |
Hi, sure. Here is the output from snap list: Name Version Rev Tracking Publisher Notes |
Can you test on 20.0.4 instead of 20.0.3 @calbav |
@SimplyCorbett gladly, can you tell me how? Would that be: $ sudo snap refresh nextcloud --channel=<what channel? latest/beta?> |
I can finally reproduce the issue on one of my tests systems. I'll look into that. |
There are basically two things we need to achieve here:
|
I have no performance issues and no reason to setup APCu or redis. If APCu will fix this issue I'll install it but I don't consider that a fix to the problem either. |
I'm manually downloading richdocumentscode_arm64.tar.gz 6.4.301 released 5h ago, it's a big package an github is slow so the download will be ready in 8h :( I installed php-apcu and changed in nextcloud config.php line I'll try tomorrow, if it will solve the perfomance problems. |
I believe @juliushaertl was simply pointing out that there is an issue today if you don't have a memcache, not trying to say that you must use APCu. You have a memcache: it's redis. |
Thanks kyrofa, you're right. Tried with richdocumentscode_arm64 6.4.203 |
I did a "sudo apt-get instal php-redis" and then "sudo snap restart nextcloud" but it did not resolve the issue. However I believe differently to SimplyCorbett that if this worked it would not be a solution. If it improves performance and allows richdocumentscode to work as it expects, this solution would be quite elegant and make the entire process more quick and snappy. Im not sure if there are some extra steps I missed to make it work. |
Snap users, please stop doing this, you already have a memcache and this will not help anything. |
Ok sorry Im still learning and doing my best :) |
Gave up and disabled |
Perhaps nor really necessary to state that I see this too. I use it in a VM with docker without root rights, I'm in the docker group. And I use 20.0.4, the latest docker image. Perhaps it helps. |
I have Ubuntu 20.04 server + Nextcloud via SNAP. Is there any tutorial on how to install Openoffice and Community Document Server? I cannot wait until the Collabora issue will be fixed. Meantime I would like to use another service for online editing of files. |
Please head over to the forum for discussing setup questions. I'll lock this conversation since we already spotted the actual issue and just need to dive into a solution as described in #1282 (comment) |
@juliushaertl isn't this solved in the meantime? :) |
No, there are still some scenarios where the requests to the code server may run longer and therefore block nextcloud requests in general. I've made an attempt to refactor the way that the connection is handled in #1901 I'll see if I can finish this soon. |
🎉🎉🎉 |
Over the last couple of weeks Snap and Docker versions of Nextcloud have updated to version 2.0.2 (snap on Friday). After the upgrade the connection to Nextcloud, particularly via the web-interface, slows to a crawl and/or times out. It appears to be caused by richdocuments/collabora.
What is strange about this is that you cannot see easily identify the problem: basic analytics show no major increase in load on the CPU, Ram or I/O, the logs don't immediately show anything out of the ordinary. If you open top/htop on the server, however, and observe for a while you can see big, temporary spikes where available CPU's are maxed out by richdocuments/the Appimage related to Collabora and the Code server. When you go to the admin settings under collabora you can see an error that says that it cannot establish a connection to the in-built CODE server.
The performance problem can be fixed by disabling both the collabora and code server apps: either in the web interface if you can get there through the performance issues, or in the server terminal (sudo snap run nextcloud.occ app:disable richdocuments ; sudo snap run nextcloud.occ app:disable richdocumentscore). After doing that normal service is resumed entirely across the Nextcloud server.
This indicates that the failure to establish a connection to the inbuilt CODE server is the underlying problem and that there is something that happens when upgrading from Nextcloud v. 2.0.1 to 2.0.2 that causes it to break. Strangely, however, this is an issue that only appears to be present in Nextcloud Snap (where slowdown occurs) and Docker (where timeouts occur) and doesn't appear to have been reported by users of the regular Nextcloud: see threads on Nextcloud Community Forum (https://help.nextcloud.com/t/nextcloud-20-0-2snap2-interface-incredibly-slow/100304), Nextcloud Snap Github (nextcloud-snap/nextcloud-snap#1569), and Nextcloud Reddit (https://www.reddit.com/r/NextCloud/comments/k7b1of/extremely_slow_interface_nextcloud_snap/).
The text was updated successfully, but these errors were encountered: