[Bug]: ABS crashes with giant ebook libraries #3259
Comments
|
Update: Still happening on 2.12.3 |
|
Is there an actual crash happening or is it getting hung up? Can you check /metadata/logs/crash_logs.txt to see if there are any crashes in there? |
|
That's what i get, some sqlite crashes on constraints: It's weird because now i forced it 2 or 3 times and it didn't get refected on crash_log, but again the server collapsed when i get into books library and the container restarted by itself. |
|
Is there anything i can do? That's making ABS tottally unusable, as my Books library is the first one to show upon app starts, Container collapses inmediatelly. Could be nice to add an option on library section to disable a concrete library instead of only deleting it :) |
Can you try using the To clarify, your server is crashing at boot or when you try to scan the library? |
Okey! I will fire an edge ABS container on my docker test machine. Will take at least 2 days to generate all metadata for that amount of books. Will report back when ready! Container is crashing with any task related to Books library, container boots fine and works until you do something with that library. |
|
Well, books, podcasts and audiobooks metadata synced really fast on my test deployment. Now with 57000 books, books library isn't making the container to crash, so seems that this fix worked really well. So go ahead, waiting patiently for the next release :D EDIT: Half an hour later, container just crashed on Books library, as i see now, just when a library scan started. That's what i saw on the log: So not sure if the container crashed or simply the socket went down. There's nothing on crash_logs.txt I'm going to follow if this thing comes up again during the weekend. EDIT2: I had a crash rear a scan regarding with a Java OOM: |
|
Thanks for helping test! Can you change your log level to Debug on the test server and then try running the scan again? That should help narrow down what is happening. |
|
So the same happened again: I am on debug mode but that's the only thing appearing related to the crash. |
advplyr
changed the title
|
You can try increasing the max memory size. To do this you pass an environment variable in your docker command. The environment variables is: Where 8192 is size in MB so you can lower that if you need to. |
|
Sorry for my clumsiness, now i saw on library settings that i set up the autoscan to work every 15 minutes accidentally this morning when i created the library. Is ABS able to abort scans if theres already one on progress? Maybe those scan overlaps were causing those OOMs. Anyways i added the env var to my compose file and im testing it, for the moment 0 crashes. Let me keep testing the container. |
|
Well, rear one and a half day of testing (no more OOMs once automatic scan each 15 mins was disabled), i didn't see any crash, i got one "Socket disconection", and the interface was not responding, but it happened only one time, researching i didn't find anything related to it on container logs and on crash_log.txt. My conclusion is: If the Edge branch didn't fully fix the issue, it probably fixed it by 99%. Another thing to take into account, general ABS is really slow performant on the UI side since i added those 57000 ebooks, it responds, but once you get into ebooks library, performance is slowish. What can be tuned for this? I understand that those are lot's of files to process inside ffmpeg for retrieving miniatures and for a SQLite database, but what can be done? Thank you :) |
|
Just an update: I can confirm that without the Java Memory bar, container doesn't crash also. 😃 |
|
Thanks for testing with a large library. We still have a ways to go in fully migrating to the new data models on sqlite (we previously used JSON files to store data). So, queries and indexes are far from optimized. I could look at your specific database to see where the slow points are and try to improve them. |
|
Glad to know! I can send you on private my DB if you want. Let's hope that helps other with big libraries also. What about let users to use 3rd party DBs like MySQL/Postgres?I know is a pain to make those type of integrities, but for big libraries maybe it's the key. Other services like for example Kavita/Jellyfin have also issues when dealing with big DBs and SQLite. Only an idea :D |
|
Yeah you can send it via Discord or email [email protected]. |
|
I also don't think the bottleneck we are hitting has to do with sqlite. Sqlite can handle much more data than that. |
Sent you a friend request on DIscord! |
|
UPDATE: Rear a week of testing (excluding the OOMs which were solved), Edge tag with those fixes is way more stable than the last release for me. Right now i cannot switch my v2.12.3 Production image to edge because i don't know which countereffects it would produce, but at least edge solved the crashes rear a week of testing. |
|
UPDATE: Deployed on production v2.13.0 and for the moment problem seems solved, even ABS performance increased a lot! Will report back here if i see that the problem reproduces :) Thanks a lot for the hard work. Most projects doesn't have all the great support we have on ABS by the devs. That's really appreciated. |
|
Update: Sadly, this afternoon, now with 2.13.0 deployed on "Production" container i got various "crashes" using the aplication and trying to open Books library. Same sympthoms as always. My monitoring throwing service down errors (Uptime Kuma, receiving alerts because the URL doesnt respond with a 200 code), app unusable throwing Socket disconnection errors / ping errors and nothing on logs. Container doesn't crash, it simply stops responding and recovers itself on 2/3 minutes. Maybe Test container doesn't have those issues because there's only one user created and online. Let me know what can i provide when you can. |
|
Can you update to 2.13.1? There was a problem with 2.13.0 which was fixed a few hours ago. |
|
Tried with 2.13.1 and 2.13.2, problem persists. |
|
Updated to 2.13.3, same thing |
|
hey! Is anything more i can debug for myself? The problem is (rear deoploying edge branch on test and latest stable on production): ABS hangs (container doesnt stop or reboot but just freezes) when opening Books library, which contains about 57000 ebooks (before adding this library it worked fine, even i have an audiobook library with 10000 audiobooks), UptimeKuma reports the service down as it doesn't report http 200 code and on about 2/3 minutes it goes live again. Sometimes it doesn't hang, but it does on a 9/10 ratio more or less. Noting appears on logs, even on container logs or crash_log, i'm on debug mode. What can i do? I love this app but for now is unusable on my case, had to go back to AntennaPod again. And again: Thanks a lot for the hard work :) |
|
Do you have periodic library scans or the watcher enabled? I would try disabling those. |
|
That was the first thing I switched off a month ago when it was reported:
|
|
Updated to 2.13.4: Issue still present. |
|
To give some more info: That's the last thing i see on logs rear the hangs and disconnections. Then ABS is tottally inresponsibe for 2 mins and recovers itself:
|
|
Hey! Any clue on this? What can i do to debug it even further? Right now it's nearly imposible to open Books library without freezing the app. Remember: 57000 files:
|
|
To give some more light on the issue, when the "hang" happens, once ABS tries to process Books library, CPU loads tends to go up, but when ABS gets unresponsive, it "hijacks" the first CPU core, with 100% load for those 2 minutes, ending to the container revovering:
Would be nice to get some feedback. Thanks :) |
|
What is the first log line after the server recovers from hanging? It may just be struggling with the large library. |
|
That's what i suspect, something makes ABS to strugle on reading ultra-large libraries. I started the listing of Books library on 21:11:45, that's the log until it recovers and lists the library: Seems that it hangs on loading authors (?) |
|
Thanks. Yeah, it looks like you're running into #3237. Series and authors are filtered to ignore ones which contain no books, such as if you have an author where a user is not able to see if their books because they're explicit or has a tag preventing them from seeing it. The queries and data model needs to be cleaned up a bit to make this work better. If you want to try manually removing the source code for the authors row on the main page, you could get rid of that. |
|
Well, it worked. Now ABS doesnt hang but at the cost of loosing a feature. Is there anything planed on edge tag for fixing this? I will be very pleased to help testing any workaround :) |
|
It may be a while before it is fully fixed. The two main things that are happening:
Could you try adding an index to the database for author creation time? That will speed up the second step (not sure if the time is mostly being spent in the first or the second step). I think you can do this by just adding a audiobookshelf/server/models/Author.js Lines 115 to 159 in 562c30c If you add the index, can you check the logs again to make sure that you're still getting 18935 authors in the log and report what the time for that query is? |
|
Nice, added the index creation to code:
Give me a couple of days to test it on edge and on stable branch. |
|
First "cold test" done on Edge:
So seems that at least at first try it doesn't make any difference, maybe because it's the first try rear creating the index, let's see in the comming days how it goes. |
|
So, in the latest hour i did some tests with the same results, even restarting the container some times, access time to books library is the same as without index implementation:
Maybe i made a mistake adding the code? |
|
Thanks for helping to test the idea. Yeah, the code edit looks correct. You can double check if the index was created by using something like https://sqlitebrowser.org/ to open the database (while ABS is not running) and the looking in the lower right corner for the indexes and see if it was created for the authors table. It should be, but an easy way to sanity check. It looks like the slow down is in step 1, which will be a lot more work to fix than just asking an index. |
|
Well, bad news then as it seems that the index was created:
|
|
🤔🤔🤔🤔🤔 |
Did that fix the problem? |
The Newest Author load time is still (v2.15) very high for my 40K Library. Smaller libraries (1-2K) are fine. `2024-10-14 08:49:38.557 2024-10-14 08:49:43.263 2024-10-14 08:49:43.672 2024-10-14 08:49:43.720 2024-10-14 08:49:47.160 2024-10-14 08:49:47.625 2024-10-14 08:52:07.970 2024-10-14 08:52:07.974 |
I think that's the correct way for temporary fixing this till the model improves. |
Well, updated to v2.15.1 and results are really promising!!!! @advplyr This: DEBUG: Loaded 20 of 57001 items for libary page in 0.45s Let me allow users to use the service again for testing it for a week to take a better conclusion. But there's a notable difference!! :D |
|
We haven't added the indexes in yet for the performance improvement so it may be a fluke that query is fast for you. We did fix the issue with db corruption you were having though. |
advplyr
added
the
awaiting release
Well, one week later now i didn't see any more crash!!! Myabe that could partially fixed it?? #3487 Anyways! It doesn't crah now, but performance will improve with new performance index. Waiting patiently for the next release :D |
|
Fixed in v2.16.0. |
github-actions
bot
removed
the
awaiting release
|
Well rear some days of testing v2.16.X i can say that performance now is FINE. I have to say also that i notice a bigger impact since this #3487 was implemented than with new indexes, anyways new indexes improves the situation a bit more :) |
What happened?
Server collapses temproary with big ebook libraries
What did you expect to happen?
ABS wotking as smooth as before adding high amounts of books.
Steps to reproduce the issue
Audiobookshelf version
v2.12.2
How are you running audiobookshelf?
Docker
What OS is your Audiobookshelf server hosted from?
Linux
If the issue is being seen in the UI, what browsers are you seeing the problem on?
None
Logs
The text was updated successfully, but these errors were encountered: