Ensure error and live_status columns are created from the start

[deldesir]

Makes identification of requested files more robust and logical for

calibre-web/cps/tasks/download.py

Lines 91 to 98 in d8a2748

	# Abort if there is not a path
	if not requested_file:
	log.info("No path found in the database")
	error = conn.execute("SELECT error, webpath FROM media WHERE error IS NOT NULL").fetchone()
	if error:
	log.error("[xklb] An error occurred while trying to download %s: %s", error[1], error[0])
	self.message = f"{error[1]} failed to download: {error[0]}"
	return

[holta]

1. What YouTube playlist URL (or any such!) is this tested with?

2. Which issues is this PR most likely to help move forward?

Possibly related:

• PR Prevent stuck videos from carrying over to next download session ["unavailable fragments" ?] #193
• PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223
• PR Increase timeout to 120s [fragment_stuck_timeout = 120 # seconds] #229

[deldesir]

I can't find the dpaste link that prompted me to propose this. I've seen in one of the logs shared there were a case of missing error column. It was a failed video redownload attempt.

[holta]

1. What YouTube playlist URL (or any such!) is this tested with?

Clarify smoke test details when you can, Thanks!

[deldesir]

Premature creation of error column is not necessary anymore because videos failed without errors are removed and reprocessed (in next retry) per #193

[holta]

Premature creation of error column is not necessary anymore because videos failed without errors are removed and reprocessed (in next retry) per #193

Interesting, Ok!

[deldesir]

I reopened this after some reexamination... Error column is expected whenever a download fails.

[holta]

Is it possible the live_status column might also be needed in advance?

• Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

[deldesir]

Is it possible the live_status column might also be needed in advance?

• Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

Yes, when a playlist URL is submitted.

[holta]

Great news if this is tested + broadly safe, or close?

[deldesir]

Smoke tested on Ubuntu 24.04. Missing columns error doesn't happen anymore

screenshot: N/A

[holta]

Smoke tested on Ubuntu 24.04. Missing columns error doesn't happen anymore

Can you test all 3 original problem URL's to confirm... ?

• (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed #186
• Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] [unavailable fragments?] #215
• Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

Are there any relevant revisions or related clarifications that can now be added at... ?

• Overall "Download to IIAB" workflow #190

[EMG70]

SUDO IIAB-DIAGNOSTICS: https://dpaste.com/GYCDC4NQG

Tested on a new VM and ran PR#191 on it.
The following video urls were tested .
1.https://www.youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM precious plastics
2. https://youtu.be/4BL65HElOPg mondiales( Nzolas's)
3. www.youtube.com/playlist?list=PLqxP5EuGxPncYWWc1cXlcuhfcoo3Wl3Ru The 5 Hardest GCSE Questions Series

RESULTS
-The first video failed to download as seen on task but did appear later in books as a successful download.This happened after triggering download of video 3 above and seems to join both playlist of 29+20 and appeared as downloading 49 videos?.
-The second video downloaded but had a recurring warning during download ( some fragments are taking longer than expected to download,please wait.The whole video eventually downloaded and warnings disappeared from tasks,see screenshot)

-The 3rd video also downloaded Ok as part of combined download albeit randomly (as part of 49 videos url 1 and url 2)
-Precious plastics videos did not create a shelf but split video authors into ( one army and precious plastics) 28/29 videos appear in books as one army and one as precious plastics.Total 29/29 downloaded OK though.This maybe due to owners naming of videos?

[holta]

@EMG70 I'm not seeing that you ran the git pull https://github.com/deldesir/calibre-web deldesir-patch-44 --no-rebase --no-edit command to install this PR — according to our usual "How do I test a pull request (PR) ?" instructions at: https://github.com/iiab/calibre-web/wiki#how-do-i-test-a-pull-request-pr-

Assuming you're trying to test this PR #191:

The branch deldesir-patch-44 should appear in your git log output, if you're in /usr/local/calibre-web-py3 — and also in your iiab-diagnostics git log output here: https://dpaste.com/GYCDC4NQG#line-1034

[holta]

@EMG70 I apologize:

Ansible was destructively removing all local commits, preventing testing of PR's !

I've now fixed that — so you can try again — testing PR #191 here if you have time:

• PR 'git pull' if upgrading Calibre-Web, bypassing Ansible's incomplete git module iiab#3777

[EMG70]

Ok will test later today.

[EMG70]

A new VM was created and ran PR#191.
SUDO IIAB-DIAGNOSTICS - https://dpaste.com/FJR5AZQEW

All three video URLs downloaded successfully.
This video ( https://youtu.be/4BL65HElOPg mondiales ( Nzolas's)) of duration 1hr56mins resulted in real time repetitive messages in tasks (some fragments are taking longer than expected to download,please wait). These messages however disappeared upon successful download.Is one message for the whole video good enough or it comes per fragment?see screenshot.

Task display after video downloaded successfully.
( https://youtu.be/4BL65HElOPg mondiales ( Nzolas's))

[deldesir]

Smoke tested on Ubuntu 24.04. Missing columns error doesn't happen anymore

Can you test all 3 original problem URL's to confirm... ?

• (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type not in session, add operation along 'Authors.books' won't proceed #186
• Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] [unavailable fragments] #215
• Metadata Fetch: [PLAYLIST URL] failed: An error occurred (no such column: live_status) while trying to connect to the database. #228

All passed (no errors), except editbooks.py: SAWarning: Object of type not in session, add operation along 'Authors.books' won't proceed #186](#186)

Are there any relevant revisions or related clarifications that can now be added at... ?

• Overall "Download to IIAB" workflow #190

No, but emphasis should be put on playlists downloads because live_status addition introduced a regression due to missing column

[deldesir]

@EMG70, thanks for testing. The repetitive messages should have been gone.

[holta]

@deldesir I suggest this be merged now, and @codewiz review it quickly for possible improvements after merging, Ok?

[deldesir]

I agree.

[holta]

@codewiz can you review this PR quick — if possible?

(Many Thanks!!)

[holta]

Somewhat related to this PR converging quickly!

• PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223