Some videos download as overweight (giant) files that (1) are not appropriate for schools, and (2) crash Calibre-Web with "502 Bad Gateway"

[holta]

EXAMPLE: The button "Download to IIAB" was used to download 23m44s video https://youtu.be/sqoOzGMqCQU which eventually downloaded into Calibre-Web (about 6min later... depending on the IIAB's Internet bandwidth of course!)

HOWEVER: Clicking either of the 2 blue buttons below [ "WEBM (4.7GB)" or "Watch in Browser - webm" ] at https://box/live/book/222 appears to cause a serious crash of... all of Calibre-Web? (or some core piece of its infra?)

QUESTION: @deldesir what is your technique for bringing NGINX and/or Calibre-Web back to life in these "502 Bad Gateway" situations? (It appears you have a technique to restart some service[s] that does not require a full system reboot?!)

CONTEXT: Just FYI this problem occurred with the latest code (from https://github.com/iiab/calibre-web) installed on a high-powered Ubuntu Server 22.04 VM with 8GB RAM, plenty of disk, and the following kernel...

# uname -a
Linux lrn2 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

[holta]

Perhaps not directly related (?) but here are 2 other/additional "502 Bad Gateway" problems also needing to be investigated and understood — on the road to robust international resiliency — in support of teachers/students worldwide living in very challenging environments: 💯

• Calibre-Web software update [upgrade] button does damage: can no longer log in, "502 Bad Gateway" #31
• "Calibre-Web needs to survive intensive requests demanding lots of RAM (e.g. playing a large video caused “502 Bad Gateway” on ALL pages with a 1GB RAM VM, crash!)"
  • Was that perhaps an isolated failure mode, arising from its under-resourced 1GB RAM Ubuntu VM's "OOM" (out of memory), possibly inducing a Linux/system-level failure?
  • Or will this also affect 2+ GB RAM VMs? (In which case a new/separate ticket may be best, to document/track this as a unique/separate problem!)

[deldesir]

This is due to excessive RAM usage. I guess implementing a handler for cases like this would be practical unless we limit big-size downloads pre-checking video formats.

[holta]

@EMG70 FYI — @deldesir is working very hard to investigate:

1. Our highest priority is to ensure large downloads (e.g. 3-hour educational documentary films) work very reliably — from Internet/Web to IIAB:

   • Setting a default of 720p downloads (to IIAB) can likely help a bit here, in some but not all cases: PR Pin max format [Set "best" <= 720p with VP9 codec for now: later consider 30FPS instead of 60FPS and WebM to tighten up further?] #48

   • Somewhat related is figuring out the accidental-but-serious overuse of disk/CPU here: 2 copies of downloaded video (unfortunately!) appear within /library #45

2. Secondarily, we need to solve reliable file downloads from IIAB to student device ("502 Bad Gateway").

3. Finally, if Calibre-Web can also make long/large videos playable in browser, that's the lowest priority (of these 3 priorities) but will hopefully be possible eventually, e.g. perhaps using this Video.js approach:

   • PR Add Video.js #49

[EMG70]

All noted. Thanks.

[deldesir]

@EMG70 and @holta, I manage to trick calibre-web to play the video externally to avoid the memory overload. See #51

[holta]

Let's keep this ticket open as work progresses around:

• PR scripts/lb-wrapper: Try yt-dlp with "--format best --format-sort 'tbr~1000'" [testing shows need for URL validation, clearly communicated, for users who accidentally request the impossible after clicking "Download to IIAB" > "Start"] #63

Much better community documentation is emerging below — telling "Grandma Gardeners" (a.k.a. Curricular Content experts) how to take the law into their own hands (installing the videos they truly need) — appropriately compressed to their needs:

• https://github.com/iiab/calibre-web/wiki#tweaking-video-quality