"Test at least 10 different Vimeo video downloads"

[holta]

@deldesir @EMG70 if you have time❔

If so please post full results ("Tasks" view screenshot and iiab-diagnostics pastebin URL) after attempting to "Download to IIAB" 10 different Vimeo videos.

🌎 🌍 🌏

Related:

• "Download to IIAB" of Vimeo video ("Staff Pick" URL or regular URL) does not download [and 1 YouTube video too, then 2 YouTube videos worked] #78
• No suitable format found for some (many?) Vimeo (and YouTube) downloads — "failed to download: 'NoneType' object is not subscriptable" #164
• 3 kinds of errors on a fresh test: (1) 'NoneType' object cannot be interpreted as an integer (2) 'NoneType' object is not subscriptable (3) Download appears to be stuck #167

[EMG70]

SUDO IIAB-DIAGNOSTICS : https://dpaste.com/BDRX23TF6
11 Vimeo videos downloads attempted:
https://vimeo.com/539844075 How feed prices will impact the broiler industry ✅
https://vimeo.com/5282155 Sewing ✅
https://vimeo.com/53301442 Cheese ✅
https://vimeo.com/228341694 Networking ✅
https://vimeo.com/953721938 Sql intro (Failed to download,non susbscriptable) ,not a calibre web problem.output message says Video is not rated. Log in to watch.? ❌ see screenshot

https://vimeo.com/498384180 IoT , video downloaded without cover ❌
https://vimeo.com/255959106 Data basics ✅
https://vimeo.com/134138657 IoT: How Data Science-Driven Software is Eating the Connected World ✅
https://vimeo.com/461859027 Beef efficiency ,video downloaded without cover ❌
https://vimeo.com/447360211 Avocado grafting ✅
https://vimeo.com/222794361 Papaya ✅

The two videos that did not show thumbnails were uploaded by SRUCBytes - Scotland's Rural College.

[holta]

THANK YOU @EMG70 for your hard work 🙏

Summary needing investigation:

8 ✅
2 ⚠️ thumbnail missing
1 ❌ "failed to download: 'NoneType' object is not subscriptable"

See also: #167 (comment)

[holta]

@EMG70 would you have time to repeat any similar test, trying to download 10 Vimeo videos and also a Vimeo showcase (i.e. playlist) ?

Reason is that related PRs have just been merged:

• PR Make error status messages user-friendly #172
• PR Simplify playlists identification [for non-YouTube playlists e.g. Vimeo showcases, Bilibili.tv / Bilibili.com anthologies, etc] #173
• PR Match playlist pattern [try to improve video downloads from ~1000 sites, not just YouTube and Vimeo — thanks to yt-dlp] #174

As included in the just-published IIAB Calibre-Web weekly release:

• https://github.com/iiab/calibre-web/releases/tag/2024.06.07

[avni]

Testing Jun 8, 2024 12pm ET - Individual Videos

iiab-diagnostics: https://dpaste.com/BBFFWJ7PQ

4 Vimeo video downloads attempted (3 from @EMG70 's list and 1 different one)

• https://vimeo.com/539844075 [How feed prices will impact the broiler industry] ✅
• https://vimeo.com/953721938 [Introduction to SQL] ❌
  • Same error as @EMG70's. Log: [2024-06-08 13:07:30,650] ERROR {cps.tasks.download:122} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
• https://vimeo.com/498384180 [SRUCBytes 3 - IoT] : ⚠️
  • Download successful, but no cover image. Same as @EMG70
• https://vimeo.com/101344613 [Montana Summer Star Trails - Night Ranch] ✅

1 YouTube video

• https://www.youtube.com/shorts/Ef3WUFCUW0w [Oregon Black bear on the move] ✅

2 Bilibili videos

• https://www.bilibili.com/video/BV1Sy4y1g7P9/?spm_id_from=333.337.search-card.all.click [Brown Bear, What do you see] ❌
  • Log: [2024-06-08 13:29:35,217] ERROR {cps.tasks.download:122} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
• https://www.bilibili.com/video/BV1i3411t7DA/?spm_id_from=333.337.search-card.all.click [Apple Round, Apple Red] ❌
  • Same error

Summary of testing:

• 7 videos, (4 Vimeo, 1 YouTube, 2 Bilibili)
• 3 ✅ successful (2 Vimeo, 1 YouTube)
• 1 ⚠️ thumbnail missing (1 Vimeo)
• 3 ❌ unsuccessful (1 Vimeo, 2 Bilibili)

More next on Playlists/Showcases/Anthologies and Channels

[holta]

• https://vimeo.com/953721938 [Introduction to SQL] ❌

  • Same error as @EMG70's. Log: [2024-06-08 13:07:30,650] ERROR {cps.tasks.download:122} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable

• https://vimeo.com/498384180 [SRUCBytes 3 - IoT] 🥉

  • Download successful, but no cover image. Same as @EMG70

@deldesir both errors above appear to be identical to @EMG70's test results 3 days prior:

Is this expected?

[deldesir]

Bilibili errors are expected due to yt-dlp/yt-dlp#10121. Vimeo ones need further investigation. I'll make an enhancement right away to read the error reported by yt-dlp (via xklb) instead of the exception at runtime.

[deldesir]

Cover for https://vimeo.com/498384180 went missing because it was placed in another directory.:

root@box:/usr/local/calibre-web-py3/cps/tasks# tree Vimeo/
Vimeo/
├── Scotland's Rural College
│   └── 498384180.jpg
└── Scotlands Rural College
    └── SRUCBytes 3 IoT_54.00_[498384180].mp4

3 directories, 2 files

[holta]

Many thanks to @chapmanjacobd for shining a light on the punctuation problem: (e.g. single quote in "Scotland's" above)

• Add option to pass/filter filenames through an arbitrary external program yt-dlp/yt-dlp#6521

• Other troublesome punctuation characters we should test:

  https://github.com/chapmanjacobd/library/blob/edb47f7a07de93e2f1d832e9a0a0142e7083f1b5/xklb/utils/strings.py#L55-L105

  https://github.com/chapmanjacobd/library/blob/edb47f7a07de93e2f1d832e9a0a0142e7083f1b5/xklb/utils/strings.py#L140-L158

[avni]

Testing Jun 8, 2024 5pm ET - YouTube Playlist

iiab-diagnostics: https://dpaste.com/HE465LXDW

I downloaded a playlist of 15 vidoes: LINK. 14 downloaded successfully, 1 had its thumbnail missing.

I liked that a Shelf for the playlist was automatically created so I can find these specific videos grouped together easily. It would be helpful to make that more clear on the Tasks page. The Tasks page has each video listed individually with no reference to the grouping.

[holta]

@avni there's an apostrophe in the title of https://youtu.be/1M22zT_w0o8 ("Switzerland's most clickbaitable hike")

▶️ This causes the thumbnail to fail to display, as reconfirmed by @deldesir, and on LRN2.ORG

▶️ Please try again in a few days with a fresh install — when this should very likely be fixed — thanks to @chapmanjacobd's hard work!

[holta]

https://youtu.be/1M22zT_w0o8 ("Switzerland's most clickbaitable hike")

ASIDE: Users are of course not thrilled that punctuation (e.g. apostrophes) are silently disappearing from original video titles...

• This is a compromise for now❕

• Possibly in future years we'll be able to respect original/full titles, including all punctuation, in all languages❔

PS the 255-character limit for PATH/FILENAME strings (in certain OS's) constrains us at the moment:

• Add option to pass/filter filenames through an arbitrary external program yt-dlp/yt-dlp#6521 (comment)

[avni]

ASIDE: Users are of course not thrilled that punctuation (e.g. apostrophes) are silently disappearing from original video titles...

Have you already tried escaping the punctuation? A quick search shows a couple of ways to escape specific characters.

[holta]

Have you already tried escaping the punctuation?

If only! Escaping would add extra characters when a book or video's PATH/FILENAME is already almost 255-characters (common OS limit) is... dangerous. Also, there's an 18-year backstory we need to respect, as much as possible, at least for now:

1. Calibre and Calibre-Web have always modified authors' names and book titles, to become somewhat "safer" (and somewhat more readable) OS folder names.

   Naturally some people aren't thrilled about Calibre's compromise/hack/kludge of "dumbing down" author names and book titles for the sake of filesystems, 18 years later (fair criticism in 2024!!) But there's no completely magical solution on the immediate horizon, as outlined in 2020 here: https://www.mobileread.com/forums/showthread.php?t=330290

2. Likewise, as outlined above, yt-dlp and xklb do something extremely similar for videos (i.e. converting every video's eccentric "original" author/title strings into a slightly safer OS PATH/FILENAME !)

Recap: There's no magic wand. As the de facto global standard is to respect /library/calibre-web/AUTHOR/TITLE for now 😇

[holta]

@EMG70 would you have time to battle-test a diversity of videos &/or playlists in coming days?

FYI a fresh install of IIAB Calibre-Web is best — as these 2 PRs were just merged:

• PR Read error from xklb-metadata.db [leveraging yt-dlp STDERR] #175
• PR Set outtmpl externally [to avoid missing thumbnails; NOTE: archival-quality book & video titles are preserved in 2 SQLite db's] #177

[EMG70]

Will do . Thanks.

[holta]

Will do . Thanks.

@EMG70's bug report:

• @EMG70's YouTube playlist downloading problems 2024-06-10: (1) (This is taking longer than expected) (2) Metadata Fetch: [YouTube video URL] failed: 'NoneType' object cannot be interpreted as an integer [@avni results: (3) failed to download: None (4) it keeps trying to redownload failed videos] #178