Enable subsequent [re]downloads of playlists #151

deldesir · 2024-03-28T02:55:17Z

Fixes #150. This PR adds a timestamp to playlists' paths after they are downloaded.

[ 2024-05-30 SUMMARY: xklb generally does not support subsequent redownloading of playlists, so let's try this workaround! ]

Fixes iiab#150, enabling subsequent [re]downloads of playlists

This prevents UNIQUE CONSTRAINT error

deldesir · 2024-03-28T03:44:22Z

Tested with subsequent [re]downloading of a small playlist. Couldn't test with long playlists due to potential disk IO error.

EMG70 · 2024-03-28T18:10:13Z

IIAB-DIAGNOSTICS - http://sprunge.us/SormHj?en

All single single videos and playlists failing to download.

https://www.youtube.com/watch?v=T3kDWpREiEI&list=PLoC8Q0moRTSjKkMZoVkAgHMJjkpUreleb&index=21e - single video
https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist
https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl - 53 short videos in playlist

deldesir · 2024-03-28T18:15:57Z

Hi @EMG70. Reading your log, it looks like xklb-metadata.db is missing. Can you check by doing ls /library/calibre-web/xklb-metadata.db using your terminal?

holta · 2024-03-28T18:26:58Z

Hi @EMG70. Reading your log, it looks like xklb-metadata.db is missing. Can you check by doing ls /library/calibre-web/xklb-metadata.db using your terminal?

Or run this as root...

apt install tree
cd /library/calibre-web
tree

EMG70 · 2024-03-28T18:32:31Z

Hi @EMG70. Reading your log, it looks like xklb-metadata.db is missing. Can you check by doing ls /library/calibre-web/xklb-metadata.db using your terminal?
Here is the output

deldesir · 2024-03-28T18:43:48Z

Line 1090 in your log says you don't have XKLB installed. How come you have the database and no XKLB installed? Weird!

deldesir · 2024-03-28T18:48:39Z

To confirm run:
which lb

If it is found, do

lb --version

EMG70 · 2024-03-28T18:49:01Z

Line 1090 in your log says you don't have XKLB installed. How come you have the database and no XKLB installed? Weird!

Allow me to reinstall and try again in case i missed something

deldesir · 2024-03-28T18:50:35Z

Please go ahead, no worry.

EMG70 · 2024-03-28T18:52:03Z

To confirm run: which lb

If it is found, do
lb --version

sorry a bit late now,VM destroyed.

EMG70 · 2024-03-28T19:51:05Z

Please check if all is ok before proceeding with testing.

deldesir · 2024-03-28T19:54:52Z

Looks good so far

holta · 2024-03-28T20:43:44Z

Plz post ideas(s) as to why the earlier XKLB install failed / disappeared, if possible?

EMG70 · 2024-03-28T22:09:35Z

IIAB-DIAGNOSTICS - http://sprunge.us/tjeC5u?en

Downloading and video ranking showing OK .Subsequent playlist downloads all failed no matter number of videos in playlist,however a single video downloads well.

This playlist downloaded all 53 videos ok but failed on second attempt. https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl
https://www.youtube.com/playlist?list=PLiMWaCMwGJXlQj0BFEwew0fvhOPr833GS 7 videos fails

deldesir · 2024-03-29T00:28:32Z

I cannot reproduce your issue @EMG70, sorry. Both Playlists work for me, all videos were downloaded, except 1 (https://www.youtube.com/watch?v=7m4_kZOObzw). I am afraid you might be subject to some restrictions either by your ISP or YouTube itself

EMG70 · 2024-03-29T00:36:06Z

You could be right.I have not done any testing since I changed ISP two weeks ago.
I will try and see what's going on.

holta · 2024-03-29T00:53:21Z

might be subject to some limitations either by your ISP or YouTube itself

We can and should arrange for @EMG70[*] to do the exact same test remotely (i.e. with a different ISP and different YouTube geographic restrictions) to confirm what exactly is happening and why.

[*] And others too, if possible!

@deldesir can you help spell out the exact steps of what most needs to be tested, to help everyone avoid ambiguity here?

deldesir · 2024-03-29T01:06:53Z

We're dealing with a download issue. We know we have a successful test if we provide 2-3 playlists one after the other and the following conditions are met for them:

metadata are fetched and total duration is displayed
videos are downloaded (a few failures are ok)
shelves are created and populated

holta · 2024-03-29T01:12:38Z

We know we have a successful test if we provide 2-3 playlists one after the other and the following conditions are met [,,,]

Thanks for explaining.

Are we also trying to test: repeatedly entering the same playlist URL?

(i.e. clicking "Download to IIAB" each time, and entering the same YouTube playlist URL each time, a few times in succession)

deldesir · 2024-03-29T06:13:37Z

I looked into t5. Everything looks Ok. Can't spot the issue right now. I will delete xklb-metadata and run the playlists again.

deldesir · 2024-04-09T14:01:10Z

Thanks a lot for testing again. The UNIQUE constraint error is a database issue. I'll let you know more once I am done examining your log.

EMG70 · 2024-04-09T14:05:26Z

Sorry did i post on wrong PR.I have just posted same on PR#150

EMG70 · 2024-04-09T14:10:27Z

Thanks a lot for testing again. The UNIQUE constraint error is a database issue. I'll let you know more once I am done examining your log.

IIAB-DIAGNOSTICS - http://sprunge.us/Kv2wBX?en
https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist ✅
https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl - 53 short videos in playlist ✅
97/102 Videos were downloaded OK.A few videos failed https://www.youtube.com/watch?v=4e3dIeP93E8 and https://www.youtube.com/watch?v=D2KvDBSZA0w although they do play well on Youtube.I am not sure why,see screenshots:

deldesir · 2024-04-11T05:39:22Z

For the 102 videos url (https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist) I also had 3 failed videos:

We have one in common: https://www.youtube.com/watch?v=4e3dIeP93E8

When I forced the redownload of any failed video I got the UNIQUE constraint failed: media.path error. It's because :
1- no error was reported by xklb for the failed video. It's like the video succeeded and even got a download/local "path"
2- since the video really failed, calibre-web did not have it, so no timestamp was added to the path

holta · 2024-04-11T13:13:22Z

Thanks @deldesir for explaining the progress:

Is targeted testing needed (e.g. from @EMG70) in coming days?

deldesir · 2024-04-11T13:30:50Z

Redownloading a playlist twice still doesn't work as expected. This will require some adjustments for sure. Hope I fix it today or tomorrow so Ed can battletest it.

deldesir · 2024-04-15T19:53:15Z

@EMG70 and @holta, I just got 100% success downloading and redownloading https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi. What changed? No code additions, I only updated my branches based on iiab:master and resolved 2 merged conflicts that led to the closing of #146. I also notice the VM where failures from redownloading happened did not have this commit d9e4fc4. Examining the [ xklb-metadata.db ] confirmed this because the downloaded playlists do not have any timestamps added to them.

http://sprunge.us/lPRkPg

holta · 2024-04-15T19:56:18Z

@deldesir what do you suggest?

deldesir · 2024-04-15T19:58:23Z

I'd like a second look into this. I want to be sure this works for @EMG70 too.

holta · 2024-04-15T20:07:19Z

Thanks @deldesir:

Please make a recommendation if additional testing is required?

deldesir · 2024-04-15T20:18:45Z

A clean test environment @holta.
@EMG70, If you plan to test this again, ensure you don't include #152. I don't understand yet what went wrong so a clean VM with only #151 is recommended. I have to acknowledge we benefit from the enhancements made recently to yt-dlp and xklb so your test will probably use:

yt-dlp 2024.04.09
xklb 2.6.009

EMG70 · 2024-04-15T23:25:40Z

I will test tomorrow and update you.

deldesir · 2024-04-16T12:04:31Z

@EMG70 and @holta, downloading channel with this PR is not solid yet due to "Shorts" not having time_uploaded. I'll let you know once I'm done handling this in a better way. For your tests, please consider playlists only for now.

EMG70 · 2024-04-16T19:58:51Z

IIAB-DIAGNOSTICS - http://sprunge.us/Rf3QgC?en

There were two failed videos ,one of them is 9hrs,not sure why the content creator included it as part of a playlist JavaScript Programming All-in-One Tutorial Series (9 HOURS!) https://www.youtube.com/watch?v=9M4XKi25I2M .
This probably explains why Task output seemed to be hung (failed to download:download appears to be stuck)

The second video that failed https://www.youtube.com/watch?v=NBRdd18N5X0 was also tried individually and still returned a failed output in Task ( failed to download:UNIQUE constraint failed.media.path).

deldesir · 2024-04-17T01:06:07Z

Happy this is getting more robust. Thanks @EMG70.

The UNIQUE constraint failed.media.path is not difficult to tackle. It's because the stuck videos are not post-timestamped. I'm looking for an elegant way to fix it 🤔.

holta · 2024-05-30T13:58:36Z

@EMG70 we're hoping a quick test with YouTube Shorts (@deldesir will help) confirms this PR can be merged quickly, ideally today!

holta · 2024-05-31T15:47:37Z

This PR was tested with #155 and #157 on t4 as explained here:

Remove residual fragments from failed video #157 (comment)

@deldesir confirms this PR #151 is ready.

Add a timestamp to playlist path after download

d9e4fc4

Fixes iiab#150, enabling subsequent [re]downloads of playlists

deldesir added the enhancement New feature or request label Mar 28, 2024

deldesir requested a review from holta March 28, 2024 02:55

deldesir self-assigned this Mar 28, 2024

deldesir marked this pull request as draft March 28, 2024 03:03

deldesir changed the title ~~Enable subsequent [re]downloads of playlists~~ Enable subsequent [re]downloads of playlists [WIP] Mar 28, 2024

Use path instead of extractor_playlist_id

33616bf

This prevents UNIQUE CONSTRAINT error

deldesir changed the title ~~Enable subsequent [re]downloads of playlists [WIP]~~ Enable subsequent [re]downloads of playlists Mar 28, 2024

deldesir marked this pull request as ready for review March 28, 2024 03:41

This was referenced Mar 28, 2024

Subsequent playlists redownloads fail #150

Closed

Ignore failed/residual/stuck video downloads: download only newly requested one(s) #147

Merged

Merge branch 'iiab:master' into deldesir-patch-25

c81254e

Merge branch 'iiab:master' into deldesir-patch-25

2c9b753

deldesir mentioned this pull request Apr 11, 2024

Videos failed and triggers "UNIQUE constraint failed: media.path" error on first retry/redownload #153

Closed

Merge branch 'master' into deldesir-patch-25

a793abe

holta mentioned this pull request Apr 16, 2024

Channel downloads fail due to residual shorts duplicates #154

Closed

deldesir mentioned this pull request Apr 30, 2024

Playlists did not fully download on newly built IIAB or upgraded server — 'NoneType' object cannot be interpreted as an integer — failed to download: Download appears to be stuck. #158

Open

holta mentioned this pull request May 30, 2024

Remove residual fragments from failed video #157

Draft

holta mentioned this pull request May 31, 2024

Remove residual [duplicate] YouTube Shorts urls from database #155

Merged

holta merged commit e35f629 into iiab:master May 31, 2024

holta mentioned this pull request Jun 11, 2024

Read error from xklb-metadata.db [leveraging yt-dlp STDERR] #175

Merged

deldesir deleted the deldesir-patch-25 branch July 1, 2024 21:39

holta mentioned this pull request Aug 26, 2024

An attempt to download [a YouTube channel] resulted in an accumulation (long list) of "unavailable videos" in Tasks, from all current and previously attempted failed downloads [why is the very same video individually downloadable?] #238

Open

Enable subsequent [re]downloads of playlists #151

Enable subsequent [re]downloads of playlists #151

Conversation

deldesir commented Mar 28, 2024 • edited by holta Loading

deldesir commented Mar 28, 2024

EMG70 commented Mar 28, 2024

deldesir commented Mar 28, 2024 • edited Loading

holta commented Mar 28, 2024

EMG70 commented Mar 28, 2024

deldesir commented Mar 28, 2024 • edited Loading

deldesir commented Mar 28, 2024

EMG70 commented Mar 28, 2024

deldesir commented Mar 28, 2024

EMG70 commented Mar 28, 2024

EMG70 commented Mar 28, 2024

deldesir commented Mar 28, 2024

holta commented Mar 28, 2024

EMG70 commented Mar 28, 2024 • edited Loading

deldesir commented Mar 29, 2024 • edited Loading

EMG70 commented Mar 29, 2024

holta commented Mar 29, 2024

deldesir commented Mar 29, 2024

holta commented Mar 29, 2024

deldesir commented Mar 29, 2024 • edited Loading

deldesir commented Apr 9, 2024

EMG70 commented Apr 9, 2024

EMG70 commented Apr 9, 2024

deldesir commented Apr 11, 2024 • edited Loading

holta commented Apr 11, 2024

deldesir commented Apr 11, 2024 • edited Loading

deldesir commented Apr 15, 2024 • edited by holta Loading

holta commented Apr 15, 2024

deldesir commented Apr 15, 2024

holta commented Apr 15, 2024

deldesir commented Apr 15, 2024 • edited Loading

EMG70 commented Apr 15, 2024

deldesir commented Apr 16, 2024

EMG70 commented Apr 16, 2024

deldesir commented Apr 17, 2024 • edited Loading

holta commented May 30, 2024

holta commented May 31, 2024 • edited Loading

deldesir commented Mar 28, 2024 •

edited by holta

Loading

deldesir commented Mar 28, 2024 •

edited

Loading

deldesir commented Mar 28, 2024 •

edited

Loading

EMG70 commented Mar 28, 2024 •

edited

Loading

deldesir commented Mar 29, 2024 •

edited

Loading

deldesir commented Mar 29, 2024 •

edited

Loading

deldesir commented Apr 11, 2024 •

edited

Loading

deldesir commented Apr 11, 2024 •

edited

Loading

deldesir commented Apr 15, 2024 •

edited by holta

Loading

deldesir commented Apr 15, 2024 •

edited

Loading

deldesir commented Apr 17, 2024 •

edited

Loading

holta commented May 31, 2024 •

edited

Loading