Playlists did not fully download on newly built IIAB or upgraded server — 'NoneType' object cannot be interpreted as an integer — failed to download: Download appears to be stuck.

[EMG70]

Short Notice from the maintainer

After 6 years of more or less intensive programming on Calibre-Web, I need a break.
The last few months, maintaining Calibre-Web has felt more like work than a hobby. I felt pressured and teased by people to solve "their" problems and merge PRs for "their" Calibre-Web.
I have turned off all notifications from Github/Discord and will now concentrate undisturbed on the development of “my” Calibre-Web over the next few weeks/months.
I will look into the issues and maybe also the PRs from time to time, but don't expect a quick response from me.

Please also have a look at our Contributing Guidelines

Describe the bug/problem
The previously working IIAB Calibre-Web download feature for playlists no longer works.Individual videos of playlist are however downloading well.Several attempts were made to reset Calibre-Web but could not download any playlist.

A clear and concise description of what the bug is. If you are asking for support, please check our Wiki if your question is already answered there.

To Reproduce

Steps to reproduce the behavior:

1. Go to 'https://www.youtube.com/playlist?list=PLX7zbg8RLwZidglkj3jNejTIShI3hXrYc'
   2.https://www.youtube.com/playlist?list=PLjqIPx3Kj-fV9Jf1Xl9zV7Ch6MoZDAL_P
2. Click on '....'
3. Scroll down to '....'
4. See error
   The download takes long irrespective of size of playlist,error says "It appears download is stuck" or failed "NonTYpe" ,object cannot be interpreted as an integer

Logfile
SUDO IIAB-DIAGNOSTICS - http://sprunge.us/r4fIkY?en

Add content of calibre-web.log file or the relevant error, try to reproduce your problem with "debug" log-level to get more output.

Expected behavior

A clear and concise description of what you expected to happen.
A successful download of a playlist is expected.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

$ hostnamectl
 Static hostname: box
       Icon name: computer-desktop
         Chassis: desktop 🖥️
      Machine ID: 70621bcb5a4640f290d82ae3fa8ff2b7
         Boot ID: 7556775add4d469ebaeaf184c3232b30
Operating System: Ubuntu 23.10                    
          Kernel: Linux 6.5.0-28-generic
    Architecture: x86-64
 Hardware Vendor: Lenovo
  Hardware Model: ThinkCentre M73
Firmware Version: FHKT50AUS

• OS: [e.g. Windows 10/Raspberry Pi OS]

• Python version: [e.g. python2.7]

• Calibre-Web version: [e.g. 0.6.8 or 087c4c5 (git rev-parse --short HEAD)]:

• Docker container: [None/LinuxServer]:

• Special Hardware: [e.g. Rasperry Pi Zero]

• Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR]

Additional context
Add any other context about the problem here. [e.g. access via reverse proxy, database background sync, special database location]

[deldesir]

Thanks @EMG70. I'll get back to you ASAP with a fix.

[deldesir]

@EMG70 Cannot reproduce your issue yet. I'd like to know your yt-dlp version if your vm is still up. Did you have a previously failed video? By the way, it looks like you didn't include PR #151 in your test (it shouldn't prevent you from downloading a playlist though). If ever you decide to give it a try again, please consider also #157, it will prevent a bug that affects redownload when a stuck video is encountered
.

[holta]

@deldesir are you saying the yt-dlp version number is not clearly visible in logs?

(If true, please fix this right away — with a separate/tiny PR ideally — thank you!)

[EMG70]

[EMG70]

All my servers do not have yt-dlp installed as a seperate package (it seems it was done as part of calibre-web)... ?

[holta]

@EMG70 can you please first run...

sudo -i

...to become root?

(Just in case yt-dlp --version needs to be run as root.)

[holta]

@deldesir please (also) confirm on a fresh VM that yt-dlp installs properly, and works, and that root can run: yt-dlp --version

(@EMG70 has confirmed that yt-dlp --version does not run for him, even as root, despite that being a fresh install just installed yesterday, as is clear from his iiab-diagnostics above.)

[holta]

FYI @EMG70 has found the symlink to yt-dlp is broken.

As his OS or environment has placed the yt-dlp program in:

/root/.local/pipx/venvs/xklb/bin/yt-dlp

Whereas the symlink points to:

/root/.local/share/pipx/venvs/xklb/bin/yt-dlp

As a result of:

https://github.com/iiab/iiab/blob/b3668e42e98b0010b571a5b8fc31b4ae45df6990/roles/calibre-web/tasks/install.yml#L77

[holta]

Just as a temporary measure, @EMG70 has forced the symlink:

[holta]

Please confirm (anybody, but especially @deldesir) if this yt-dlp path problem is a difference between Ubuntu 23.10 and 24.04 ?

(And if so, which other OS's have this path problem?)

[deldesir]

I confirm yt-dlp --version works on a fresh IIAB on ubuntu 24.04.

[holta]

The yt-dlp path needs to be verified on these OS's:

• Ubuntu 22.04 (or Mint 21)
• Debian 12
• Raspberry Pi OS

[holta]

Can someone please verify this very simple path question on Debian 12 ?

[holta]

@deldesir can you help @EMG70 with his original question:

• Even after having forced the correct yt-dlp symlink path on Ubuntu 23.10, @EMG70 still cannot download playlists.
• What should he try next on 23.10?
• Please assist him by doing your own installation on any 23.10 VM to validate, Thanks!

[holta]

Can someone please verify this very simple path question on Debian 12 please?

Just for the record, IIAB changed to the longer /root/.local/share path on Dec 30, again as a convenience for implementers who want to run yt-dlp at the CLI:

iiab/iiab@624827e

[EMG70]

I will try on Debian 12 and update you.

[EMG70]

IIAB-DIAGNOSTICS -http://sprunge.us/ah4JJU?en
yt-dlp not found on a fresh debian 12 iiab install .
Playlist download going well 28/40 videos so far downloaded ok.
https://www.youtube.com/playlist?list=PLX7zbg8RLwZidglkj3jNejTIShI3hXrYc

[EMG70]

I confirm yt-dlp --version works on a fresh IIAB on ubuntu 24.04.

Confirmed my side as well.It seems the problem is on my 23.10 PC which i was trying with.

[holta]

Thanks @EMG70:

While running yt-dlp at the command line is really just a convenience and unrelated to the core #158 problem you've reported, I'll make yt-dlp symlink creation more resilient on older OS's, to make this more clear:

• PR Safer /usr/local/bin/yt-dlp symlink creation iiab#3732

PS Just a reminder for all that Debian 12 VM's can instantly be created as follows:

multipass launch https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2

Of course VM needs to have sufficient cores/RAM/disk and networking...

multipass launch https://cloud.debian.org/images/cloud/bookworm/daily/latest/debian-12-generic-amd64-daily.qcow2 -n deb12 -c 2 -m 2G -d 20G --bridged

[holta]

@EMG70 are you sure Ubuntu 23.10 is a problem?

It would seem to be proceeding ( to download many-if-not-all videos from playlist https://www.youtube.com/playlist?list=PLX7zbg8RLwZidglkj3jNejTIShI3hXrYc ) for me...

iiab-diagnostics: http://sprunge.us/KvdN0j?en

root@box:~# yt-dlp --version
2024.04.09
root@box:~# ls -l /usr/local/bin/yt-dlp
lrwxrwxrwx 1 root root 39 Apr 30 19:32 /usr/local/bin/yt-dlp -> /root/.local/pipx/venvs/xklb/bin/yt-dlp

[holta]

@deldesir please help everyone understand why about 1/3 of videos failed to download above.

Specifically, here are 12 of 40 videos that failed:

Download: https://www.youtube.com/watch?v=W8kGszi_3S0 failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=hgZpRRkh0xc failed to download: Download appears to be stuck.
Download: https://www.youtube.com/watch?v=SWDWAcjfGKU failed to download: Download appears to be stuck.
Download: https://www.youtube.com/watch?v=0I86fPLIXiM failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=H6iYoin0vIg failed to download: Download appears to be stuck.
Download: https://www.youtube.com/watch?v=flx5CAbiUPo failed to download: Download appears to be stuck.
Download: https://www.youtube.com/watch?v=QJVAK4Ut99g failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=CTunZ5hJEEM failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=-BjQJ3LDauE failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=UC5xkeaQVRo failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=Rkk42p3MY4U failed to download: Download appears to be stuck.

Download: https://www.youtube.com/watch?v=ZmPQPmF9RLU failed to download: Download appears to be stuck.

[deldesir]

@holta these failures are related to #135. The affected videos just cannot be downloaded due to unavailable fragments.

Trying to understand this issue, I noticed last month all yt-dlp versions do not behave the same.

For example this video downloaded or failed based on the underlying yt-dlp version.

https://www.youtube.com/watch?v=fAT52n5Lavg

YT-DLP 2023.11.16: ✅

YT-DLP 2024.03.10: ❌

[holta]

For example this video downloaded or failed based on the underlying yt-dlp version.

https://www.youtube.com/watch?v=fAT52n5Lavg

YT-DLP 2023.11.16: ✅

YT-DLP 2024.03.10: ❌

@deldesir that's a powerful discovery.

Can you post a ticket to https://github.com/yt-dlp/yt-dlp/issues explaining this yt-dlp regression for all?

Ideally with a minimal reproducer CLI example, e.g. an actual yt-dlp --print command as outlined by JC below? (from Apr 16)

if you can't reproduce it using only yt-dlp --print then make a GitHub PR in the xklb repo which adds a test

you can run pytest like this:

pytest --pdb tests/test_lb.py -k test_function

see tests/createdb/test_tube_add.py for an example

[holta]

Documentation https://github.com/iiab/calibre-web/wiki#gear-back-end clarified for @EMG70 and implementers/contributors:

[EMG70]

@EMG70 are you sure Ubuntu 23.10 is a problem?

It would seem to proceeding ( to download many-if-not-all videos from playlist https://www.youtube.com/playlist?list=PLX7zbg8RLwZidglkj3jNejTIShI3hXrYc ) for me...

iiab-diagnostics: http://sprunge.us/KvdN0j?en

root@box:~# yt-dlp --version
2024.04.09
root@box:~# ls -l /usr/local/bin/yt-dlp
lrwxrwxrwx 1 root root 39 Apr 30 19:32 /usr/local/bin/yt-dlp -> /root/.local/pipx/venvs/xklb/bin/yt-dlp

                                                          RESPONSE TO ABOVE 

SUDO IIAB-DIAGNOSTICS - http://sprunge.us/V8a9MF?en

I have created a new 23.10 VM and tested as follows.I can now download playlists ,35/40 downloaded OK - https://www.youtube.com/playlist?list=PLX7zbg8RLwZidglkj3jNejTIShI3hXrYc .The 5 failed videos had error ( failed to download:download appears to be stuck)

I have no idea why a PC running 23.10 with Calibre-Web upgraded stopped working as before.

[holta]

I have created a new 23.10 VM and tested as follows.I can now download playlists ,35/40 downloaded OK - https://www.youtube.com/playlist?list=PLX7zbg8RLwZidglkj3jNejTIShI3hXrYc .The 5 failed videos had error ( failed to download:download appears to be stuck)

Thank you @EMG70 for another very valuable data point !

Conclusion: Ubuntu 23.10 is not the problem (especially now that PR iiab/iiab#3732 is merged!)

[holta]

@nzola is also experiencing a lot of "failed to download: Download appears to be stuck." errors.

Is it possibly happening a lot more often among longer videos of 60+ min duration?

(Please confirm if you see a distinct pattern!)

[deldesir]

Improvements have been made since. The flash message "This is taking longer than expected" is not an error message. It showed up when no data is received for 30 seconds or more. It generally resumes back to normal. Be aware network issue can trigger this too.

[holta]

@EMG70 please retest downloading the above 12 videos if you can?!

[EMG70]

@EMG70 please retest downloading the above 12 videos if you can?

SUDO IIAB-DIAGNOSTICS - https://dpaste.com/7RJ2958HS

I have tried to download the whole playlist Bricklaying for beginners
[Rodian Builds] (https://www.youtube.com/@RodianBuilds) .It still has many errors,see screenshot.

I have also tried to download as individual videos but still getting 500 internal server error.The realtime warning (video appears stuck no longer appears0,Infact has not appeared since timer was upgraded recently.

[holta]

@deldesir please investigate on Monday, can you?

🙏