[Bug]: New library scan crashes server #2750

szethh · 2024-03-15T22:16:49Z

Describe the issue

This is on a fresh install of audiobookshelf.
I added a new library for ebooks. I pointed the folder to my calibre data location and started the first scan. This maxes out the ram and eventually the server becomes unresponsive and I have to reboot the VM.

At first I tried with 4GB of RAM, and it crashed. I gave it 8GB and it crashed again. This is with around ~150 ebooks, so nothing big (5.4GB in total).

I have seen similar issues where the culprit is ffprobe spawning without limit, but it seems it is not the case here. All the ram is taken up by the node process, only 1/2 instances of ffprobe are running.

Steps to reproduce the issue

Install the latest docker audiobookshelf image
Add a book library, point to a folder with books
Start a scan

Audiobookshelf version

v2.8.0

How are you running audiobookshelf?

Docker

advplyr · 2024-03-15T22:27:09Z

What are the file formats of the ebooks you are scanning in? If you check the scan log is there a place where it stops that we can see?

szethh · 2024-03-15T22:32:54Z

These are the logs from the container:

audiobookshelf  | [2024-03-15 22:29:41.339] INFO: === Starting Server ===
audiobookshelf  | [2024-03-15 22:29:41.544] INFO: [Server] Init v2.8.0
audiobookshelf  | [2024-03-15 22:29:41.982] INFO: [Database] Initializing db at "/config/absdatabase.sqlite"
audiobookshelf  | [2024-03-15 22:29:42.447] INFO: [Database] Db connection was successful
audiobookshelf  | [2024-03-15 22:29:43.673] INFO: [Database] Db initialized with models: user, library, libraryFolder, book, podcast, podcastEpisode, libraryItem, mediaProgress, series, bookSeries, author, bookAuthor, collection, collectionBook, playlist, playlistMediaItem, device, playbackSession, feed, feedEpisode, setting, customMetadataProvider
audiobookshelf  | [2024-03-15 22:29:44.038] INFO: [LogManager] Init current daily log filename: 2024-03-15.txt
audiobookshelf  | [2024-03-15 22:29:44.168] INFO: [BackupManager] 0 Backups Found
audiobookshelf  | [2024-03-15 22:29:44.169] INFO: [BackupManager] Auto Backups are disabled
audiobookshelf  | [2024-03-15 22:29:44.358] INFO: [Watcher] Initializing watcher for "Books".
audiobookshelf  | Warning: connect.session() MemoryStore is not
audiobookshelf  | designed for a production environment, as it will leak
audiobookshelf  | memory, and will not scale past a single process.
audiobookshelf  | [2024-03-15 22:29:44.471] INFO: Listening on port :80
audiobookshelf  | [2024-03-15 22:29:45.247] INFO: [Watcher] "Books" Ready
audiobookshelf  | [2024-03-15 22:29:45.938] INFO: [SocketAuthority] Socket Connected Dnmu_xhTgeLU9OEFAAAB
audiobookshelf  | [2024-03-15 22:30:02.373] INFO: [SocketAuthority] Socket Dnmu_xhTgeLU9OEFAAAB disconnected from client "root" after 16435ms (Reason: transport close)
audiobookshelf  | [2024-03-15 22:30:03.637] INFO: [SocketAuthority] Socket Connected 243an9Uks3ISqBIRAAAD
audiobookshelf  | [2024-03-15 22:30:04.939] INFO: [LibraryScanner] Library metadata precedence changed since last scan. From [Unset] to [folderStructure,audioMetatags,nfoFile,txtFiles,opfFile,absMetadata]
audiobookshelf  | [2024-03-15 22:30:04.941] INFO: [LibraryScanner] Starting (forced) library scan 206d37a8-2505-43a9-a924-a28b3855abc2 for Books
audiobookshelf  | [
audiobookshelf  |   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
audiobookshelf  | ]
audiobookshelf  | [ 'falling back to ArrayBuffer instantiation' ]
audiobookshelf  | [
audiobookshelf  |   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
audiobookshelf  | ]
audiobookshelf  | [ 'falling back to ArrayBuffer instantiation' ]
...

That last part keeps repeating until I kill the process

szethh · 2024-03-15T22:55:25Z

also, regarding the file formats:

szeth@kite:/mnt/storage/media/books/calibre$ find . -type f | awk -F. '{print $NF}' | sort | uniq -c
      7 DS_Store
      4 azw3
      3 cbr
     10 cbz
      2 db
      2 djvu
     93 epub
     82 jpg
      1 json
     39 mobi
      3 mp3
     82 opf
     28 pdf

The .db are calibre databases. The json is also for calibre. So it's mostly epub, mobi, pdf and azw3. The jpgs are covers, opf is calibre book metadata. Some comics too (cbr, cbz).

Is any of these formats known to cause issues?

advplyr · 2024-03-15T23:28:56Z

I think the cbr comics are having issues right now I'm still investigating. Can you take those out for a test scan?

szethh · 2024-03-16T00:06:04Z

yup that fixes it. both cbr and cbz seem to be causing issues.

one thing to note though, they do seem to be getting partially processed. If i start a scan with a cbr/cbz file present, it goes into the wasm error loop. if i then move the files out of the directory, this shows up in the logs:

audiobookshelf  | [
audiobookshelf  |   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
audiobookshelf  | ]
audiobookshelf  | [ 'falling back to ArrayBuffer instantiation' ]
audiobookshelf  | [2024-03-16 00:03:07.718] ERROR: [FileUtils] ReadTextFile error Error: ENOENT: no such file or directory, open '/storage/media/books/calibre/Tatsuki Fujimoto/Chainsaw Man 04 (83)/metadata.opf' (fileUtils.js:126)
audiobookshelf  | [2024-03-16 00:03:07.741] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 03" - marking as missing
audiobookshelf  | [2024-03-16 00:03:07.822] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 05" - marking as missing
audiobookshelf  | [2024-03-16 00:03:07.890] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 05" - marking as missing
audiobookshelf  | [2024-03-16 00:03:10.205] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 01" - marking as missing
audiobookshelf  | [2024-03-16 00:03:10.331] INFO: [LibraryItem] Library item "cb5d548e-4767-47e2-9957-e186275d4ce3" updated
audiobookshelf  | [2024-03-16 00:03:10.342] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 01" - marking as missing
audiobookshelf  | [2024-03-16 00:03:10.458] INFO: [LibraryItem] Library item "428b0783-4d28-48f0-9fa3-dc28cddb2d7e" updated
audiobookshelf  | [2024-03-16 00:03:10.463] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 02" - marking as missing
audiobookshelf  | [2024-03-16 00:03:10.980] INFO: [LibraryItem] Library item "1a54e417-dbcb-4a1b-9c6b-65da5c4376d6" updated
audiobookshelf  | [2024-03-16 00:03:10.987] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 02" - marking as missing
audiobookshelf  | [2024-03-16 00:03:11.145] INFO: [LibraryItem] Library item "7da281b9-de93-4c96-a113-6770eac6bbd8" updated
audiobookshelf  | [2024-03-16 00:03:11.159] INFO: [LibraryScanner] Scanning file update group and library item was deleted "Chainsaw Man 03" - marking as missing
audiobookshelf  | [2024-03-16 00:03:11.272] INFO: [LibraryItem] Library item "638fb745-307b-414c-8686-8cb2526d0bcd" updated

So they do seem to be somewhat processed. I do not know much about this project so no clue why this is 😅

Maran123 · 2024-03-20T10:40:44Z

I think i'm also encountering this issue with cbz files. It works if you have enough RAM and/or swap-space for the entire library of cbz files. Otherwise the memory fills up and the scan is slow/halted.

szethh · 2024-03-20T10:50:56Z

this seems unfeasible for moderately big libraries.

are all book files buffered in memory at once?? 😬

also, does this happen with audiobooks too? or just cbz?

nichwall · 2024-03-20T13:28:10Z

It's only with some cbz/cbr and ebook files.

Tsamsiyu1 · 2024-03-20T19:07:59Z

i have the same problem. i just created a new Library and added about 40 cbz files. the first 20 where scanned but it took ages. after that it just crashes the server.

maxcovergold · 2024-04-11T09:46:51Z

Tested adding comics for the first time and ran into this issue. System froze then crashed. Tried creating a small <1GB library and that completed, but noticed memory wasn't released on completion of successful scan. ABS required restart to resolve.

I also see the below error, both when memory completely fills and crashes and when scaning a small library so no crash, but memory is not released:

'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'

EDIT - I subsequently also tried adding a couple of issues at a time after successfully scanning in and restarting the ABS container to release memory, but after a couple of times, hit the ceiling where it would crash again. So files already scanned into library also contribute to the memory usage issue on new scan. Which means cannot add to library incrementally either.

maxcovergold · 2024-04-23T17:05:35Z

Unfortunately fixing of error #2856 doesn't resolve this issue. It possibly is stopping the application from crashing, but just makes my entire Synology grind to a halt indefinitely

Codex966 · 2024-06-11T16:45:49Z

Still running into this issue as well.
I added a Comics library last week, primarily with pdf, cbr, and cbz files in it and once I got to about 20~ items any scan crashes the app, requiring a restart (running Docker on UnRaid).

I've tried removing new entries to back-track if there might be a corrupt cbz file causing it, but haven't pinpointed a specific file for it yet.
My ebook/audiobook library is about 2 years old and is up to about 600 items and has no problems. Still think it might be something with how ABS is trying to scan these cbz/cbr.
(Not sure about the RAM troubleshooting though, I don't see an abnormal uptick in usage when scanning (aside from a normal amount) and I've got 128Gb of RAM on this server, my Comics library only being 13Gb so far.)

pekempy · 2024-08-05T11:07:06Z

I also have this issue with .cbr files.

I've noticed that 64/75 of The Sandman have processed properly, but it's having issues with the same sets of files after I delete and re-create the library (issue 1, issues 10-19). I've confirmed those files are not corrupted and have no differences to the other issues.

Docker logs:

2024-08-05 12:04:13   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:13 ]
2024-08-05 12:04:13 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:16 [
2024-08-05 12:04:16   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:16 ]
2024-08-05 12:04:16 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:19 [
2024-08-05 12:04:19   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:19 ]
2024-08-05 12:04:19 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:22 [
2024-08-05 12:04:22   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:22 ]
2024-08-05 12:04:22 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:25 [
2024-08-05 12:04:25   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:25 ]
2024-08-05 12:04:26 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:29 [
2024-08-05 12:04:29   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:29 ]
2024-08-05 12:04:29 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:32 [
2024-08-05 12:04:32   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:32 ]
2024-08-05 12:04:32 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:33 [
2024-08-05 12:04:33   'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'
2024-08-05 12:04:33 ]
2024-08-05 12:04:33 [ 'falling back to ArrayBuffer instantiation' ]
2024-08-05 12:04:22 [2024-08-05 11:04:22.352] INFO: [SocketAuthority] Socket Connected Z-sfFta0-vKtwBH6AAAF
2024-08-05 12:04:38 Config /config /metadata
2024-08-05 12:04:38 [2024-08-05 11:04:38.824] INFO: === Starting Server ===

pierrejo · 2024-08-06T09:29:37Z

Hi, i've the same issue. I'm on a raspberry pi with 4Gb of RAM. And i can't add my CBZ files.

I'm on the V2.12.1 of Audiobookshelf

Error :

wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'

I've also try to limit my container with :

    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: "512M" 
        reservations:
          cpus: "0.5"
          memory: "256M"

Still get the error .... 😒

danielb2 · 2024-08-07T01:41:42Z

I'm also having the same issues with cbr files:

'wasm streaming compile failed: TypeError: Failed to parse URL from /client/dist/libarchive/wasm-gen/libarchive.wasm'

Init v2.12.1
Node.js Version: v20.16.0
Platform: linux
Arch: x64

Atfynt · 2024-08-08T20:29:44Z

It looks like the issue is in libarchive.js preventing it from automatically closing the archive once it has been opened adding archive.close() wherever we have Archive.open() just before returning would prevent having everything in memory all at the same time. at the very least parseComicMetadata.js parse function and extractCoverImage. it probably also needs to be added wherever the comic reader is extracting files but I have not really looked into that.

nichwall · 2024-08-20T05:17:18Z

It looks like the issue is in libarchive.js preventing it from automatically closing the archive once it has been opened adding archive.close() wherever we have Archive.open() just before returning would prevent having everything in memory all at the same time. at the very least parseComicMetadata.js parse function and extractCoverImage. it probably also needs to be added wherever the comic reader is extracting files but I have not really looked into that.

Thanks for helping track that down. I attempted a fix here #3312 for the server crashing during a scan if you want to help test it (I don't actually have any comic files and am testing with homemade CBR/CBZ).

advplyr · 2024-11-22T23:26:02Z

Is this still an issue?

danielb2 · 2024-11-23T14:48:47Z

works for me 👍

another issue now that I can read it is that i can't zoom out to see all of it, but I have to scroll a lot

danielb2 · 2024-11-23T14:51:18Z

also seeing this for some

szethh added the bug Something isn't working label Mar 15, 2024

nichwall mentioned this issue Apr 16, 2024

[Bug]: Bad epub causes crash with Unhandled rejection error #2856

Closed

nichwall mentioned this issue Aug 10, 2024

[Bug]: ABS crashes with giant ebook libraries #3259

Closed

nichwall mentioned this issue Aug 20, 2024

Close comics during scan #3312

Merged

advplyr added the waiting Waiting for OP label Nov 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: New library scan crashes server #2750

[Bug]: New library scan crashes server #2750

szethh commented Mar 15, 2024

advplyr commented Mar 15, 2024

szethh commented Mar 15, 2024

szethh commented Mar 15, 2024

advplyr commented Mar 15, 2024

szethh commented Mar 16, 2024

Maran123 commented Mar 20, 2024

szethh commented Mar 20, 2024

nichwall commented Mar 20, 2024 •

edited

Loading

Tsamsiyu1 commented Mar 20, 2024

maxcovergold commented Apr 11, 2024 •

edited

Loading

maxcovergold commented Apr 23, 2024

Codex966 commented Jun 11, 2024 •

edited

Loading

pekempy commented Aug 5, 2024 •

edited

Loading

pierrejo commented Aug 6, 2024 •

edited

Loading

danielb2 commented Aug 7, 2024

Atfynt commented Aug 8, 2024

nichwall commented Aug 20, 2024

advplyr commented Nov 22, 2024

danielb2 commented Nov 23, 2024

danielb2 commented Nov 23, 2024

[Bug]: New library scan crashes server #2750

[Bug]: New library scan crashes server #2750

Comments

szethh commented Mar 15, 2024

Describe the issue

Steps to reproduce the issue

Audiobookshelf version

How are you running audiobookshelf?

advplyr commented Mar 15, 2024

szethh commented Mar 15, 2024

szethh commented Mar 15, 2024

advplyr commented Mar 15, 2024

szethh commented Mar 16, 2024

Maran123 commented Mar 20, 2024

szethh commented Mar 20, 2024

nichwall commented Mar 20, 2024 • edited Loading

Tsamsiyu1 commented Mar 20, 2024

maxcovergold commented Apr 11, 2024 • edited Loading

maxcovergold commented Apr 23, 2024

Codex966 commented Jun 11, 2024 • edited Loading

pekempy commented Aug 5, 2024 • edited Loading

pierrejo commented Aug 6, 2024 • edited Loading

danielb2 commented Aug 7, 2024

Atfynt commented Aug 8, 2024

nichwall commented Aug 20, 2024

advplyr commented Nov 22, 2024

danielb2 commented Nov 23, 2024

danielb2 commented Nov 23, 2024

nichwall commented Mar 20, 2024 •

edited

Loading

maxcovergold commented Apr 11, 2024 •

edited

Loading

Codex966 commented Jun 11, 2024 •

edited

Loading

pekempy commented Aug 5, 2024 •

edited

Loading

pierrejo commented Aug 6, 2024 •

edited

Loading