DB Update Performance Issue with large library

[kevin71246]

I installed the Spivak app, pointed to my Karaoke collection "/media/pi/LinuxWindow/Karaoke", which is a separate exFAT partition consisting of 27 folders "#,A...Z", with corresponding files inside, and attempted to UPDATE the database/library. "-" is my separater BTW. Immediately after clicking the UPDATE button, it shows the message "No such file or directory" under "Collection Management" in the GUI, next to the STOP button. I didn't know if this was fatal, or if it's even doing anything. Upon opening up the web interface, I can see songs being added, but at an EXTREMELY slow pace. My collection is about ~100k files (~50k songs). Format is cdg & a corresponding mp3 file. It's been 10 hours(!) and only the #, A, B, & C folders are showing up. At this rate it'll take over 3 days to complete!

Some things I noticed, once I selected "/media/pi/LinuxWindow/Karaoke" as my library location, it changed it in the GUI to "/run/user/1000/doc/36b386e5/Karaoke". Sure enough this folder exists with the same directory structure & files as my collection. Why is it copying these, instead of making a database? Seems strange. I have 1 total SSD HDD and a Raspberry Pi 4b 8GB, with the Raspberry Pi Linux11/Bullseye OS (50GB) & this additional exFAT partition (~850GB). I also have OpenKJ installed; setting up that database with the same collection takes 2 minutes! I do however like the Spivak web interface very much, so would like to see this resolved and use Spivak instead of OpenKJ. I did test Spivak initally with a tiny collection of 25 songs and it updates instantly (<1second) and works great. I installed this on a fresh OS install via Flathub.

Could there be special characters or issues with my collection? Or is the way the app is setting up the collection not compatible with a separate partition, or exFAT? Or an OS config I can do to fix this, as opposed to addressing this on the app? Any suggestions? Thank you for all your hard work!!

[xcelente]

I've only just come across this app and can confirm this is a big issue. I have similar problems and have failed to create any kind of database. Having read this, I don't want to duplicate the files I already have. My collection is 7TB, more than 2.2 million files.

Anybody know of an alternative for linux?

[gyunaev]

To avoid scanning, you can scan the files yourself and create a file named index.dat in the collection root folder. This file should have the following format:

<artist>|<title>|<lyrics path from collection root>|<music path from collection root>|<type>|<language>

i.e. something like:

Dire Straits|Sultans of Swing|D/DI/Dire Straits/Sultans of Swing.cdg|D/DI/Dire Straits/Sultans of Swing.mp3|cdg|en

If this file is found, the scanner will just read/download this file and use info from it when scanning.

See SongDatabaseScanner::parseCollectionIndex() for further info.