(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed

[holta]

22 of 29 videos downloaded for @EMG70 (76% is a start, but also not that great!) when he tried to download...

• "Precious Plastic Academy"
  https://www.youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM

Two main issues appear to be:

1. download.py: sqlite3.OperationalError: no such column: error after TypeError: 'NoneType' object is not subscriptable (appeared just once?)
2. editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed (appears often!)

Excerpt from his iiab-diagnostics https://dpaste.com/C8SLESTWU showing both:

-rw-r--r-- 1 root root 94883 Jun 14 17:34 /var/log/calibre-web.log
                        ...ITS LAST 100 LINES FOLLOW...

[2024-06-14 17:31:52,496]  INFO {cps.tasks.download:106} Successfully sent the requested file to http://192.168.0.28/books/meta
[2024-06-14 17:31:52,509]  INFO {cps.tasks.download:129} Download task for https://www.youtube.com/watch?v=NuYj4hDjKzU completed successfully
[2024-06-14 17:31:52,509]  INFO {cps.tasks.download:41} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=w6Hyd9c5QMw']
[2024-06-14 17:32:13,810]  INFO {cps.editbooks:385} Received metadata request: ImmutableMultiDict([('requested_file', '/library/downloads/calibre-web/Youtube/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0)_35.80k_[w6Hyd9c5QMw].mp4'), ('current_user_name', 'Admin'), ('shelf_id', '4')])
[2024-06-14 17:32:13,810]  INFO {cps.editbooks:387} Requested file: /library/downloads/calibre-web/Youtube/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0)_35.80k_[w6Hyd9c5QMw].mp4
[2024-06-14 17:32:13,811]  INFO {cps.editbooks:392} Processing file: <_io.BufferedReader name='/library/downloads/calibre-web/Youtube/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0)_35.80k_[w6Hyd9c5QMw].mp4'>
[2024-06-14 17:32:13,811] DEBUG {cps.uploader:376} Temporary file: /tmp/calibre_web/84b88f2fea2e0d667acc017fdc57c534
[2024-06-14 17:32:13,821]  WARN {py.warnings:110} /usr/local/calibre-web-py3/cps/editbooks.py:1535: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed (This warning originated from the Session 'autoflush' process, which was invoked automatically in response to a user-initiated operation.)
  db_element = db_session.query(db_object).filter((func.lower(db_filter).ilike(add_element))).first()

[2024-06-14 17:32:13,832] DEBUG {cps.helper:548} Moving title: /tmp/calibre_web/84b88f2fea2e0d667acc017fdc57c534 to /library/calibre-web/One Army/Precious Plastic - Start a business with plastic recycling (part 6.0) (29)/Precious Plastic - Start a business with plastic recycling (part 6.0) - One Army
[2024-06-14 17:32:13,873]  INFO {cps.tasks.download:106} Successfully sent the requested file to http://192.168.0.28/books/meta
[2024-06-14 17:32:13,887]  INFO {cps.tasks.download:129} Download task for https://www.youtube.com/watch?v=w6Hyd9c5QMw completed successfully
[2024-06-14 17:32:13,887]  INFO {cps.tasks.download:41} Subprocess args: ['lb-wrapper', 'dl', 'https://www.youtube.com/watch?v=BtrfTSSGHEo']
[2024-06-14 17:32:28,072] ERROR {cps.tasks.download:122} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-06-14 17:32:28,073] ERROR {cps.services.worker:202} no such column: error
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 89, in run
    requested_file = conn.execute("SELECT path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/services/worker.py", line 199, in start
    self.run(*args)
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 123, in run
    self.message = f"{self.media_url_link} failed to download: {self.read_error_from_database()}"
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 139, in read_error_from_database
    error = conn.execute("SELECT error FROM media WHERE webpath = ?", (self.media_url,)).fetchone()[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: error

His screenshot:

[holta]

@deldesir

2. SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed (appears often!)

Do the notes & "potential fix" below from @codewiz help with the (SQLAlchemy) SAWarning?

https://chatgpt.com/share/1c5737a6-e9fc-4eee-83e2-9aff755ed0a2

[holta]

Thanks Much @deldesir for posting your ideas here when you can!

[holta]

@deldesir please try to reproduce both errors this week if you have a moment?🕯️

[holta]

@deldesir is this (#186) still an issue?

(Or is it possibly/definitely resolved?)

Possibly related:

• PR Ensure error and live_status columns are created from the start #191
• PR Prevent stuck videos from carrying over to next download session ["unavailable fragments" ?] #193

[deldesir]

I cannot reproduce this issue. I cannot spot the error messages in my tests.

[holta]

@EMG70 can you please re-test?

• 18+ days ago your downloading of playlist "Precious Plastic Academy"
  https://youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM only worked with 22 of 29 videos!

• Does a fresh install of IIAB Calibre-Web now download all 29 of its videos ?

[EMG70]

I will test today and update you.

[EMG70]

Tested on a new VM
SUDO IIAB-DIAGNOSTICS -https://dpaste.com/BS87LW36V

Playlist https://www.youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM 24/29 Videos successfully downloaded .


The 5 failed videos were attempted individually and all failed as well.

[holta]

24/29 Videos successfully downloaded

24 is better than 22 (from 3 weeks ago!)

• @deldesir the ~5 errors in /var/log/calibre-web.log (same for each failed video?) would appear to be exactly as (1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable" in the subject line of this ticket.
• For example starting on Line 1182 for https://youtu.be/dP1s7viFZHY pasted in below.
• Please help explain what appears to be happening — Thank You.

(Whereas @EMG70 no longer experienced (2) editbooks.py: SAWarning: Object of type <Books> not in session, add operation along 'Authors.books' won't proceed in the subject line of this ticket. Did PR #193 really solve that aspect?)

[2024-07-02 19:14:48,096] ERROR {cps.tasks.download:128} An error occurred during the subprocess execution: 'NoneType' object is not subscriptable
[2024-07-02 19:14:48,096] ERROR {cps.services.worker:202} no such column: error
Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 89, in run
    requested_file = conn.execute("SELECT path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()[0]
                     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
TypeError: 'NoneType' object is not subscriptable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/calibre-web-py3/cps/services/worker.py", line 199, in start
    self.run(*args)
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 129, in run
    self.message = f"{self.media_url_link} failed to download: {self.read_error_from_database()}"
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/calibre-web-py3/cps/tasks/download.py", line 145, in read_error_from_database
    error = conn.execute("SELECT error FROM media WHERE webpath = ?", (self.media_url,)).fetchone()[0]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
sqlite3.OperationalError: no such column: error

[holta]

Thanks to @nzola for uncovering another example here:

• Formerly live https://youtu.be/4BL65HElOPg fails to download [same "no such column: error" issue as #186 ?] [unavailable fragments?] #215

[holta]

(1) download.py: "sqlite3.OperationalError: no such column: error" after "TypeError: 'NoneType' object is not subscriptable"

@deldesir believes that "TypeError: 'NoneType' object is not subscriptable" errors almost always refer to missing data in xlkb-metadata.py for example:

• missing column error or live_status
• video duration 0 (zero) or non-existent, see:

  calibre-web/cps/tasks/metadata_extract.py

  Line 74 in 486b7bb

  if duration is not None and duration > 0:

And that improvements in the last 2 months to clearly communicate those errors in "Tasks" view should now prevent this problem from happening (e.g. I just successfully downloaded playlist https://youtube.com/playlist?list=PLXpbdfI05JvRT07cgyZjwcfSYGiFgnbtM and its 29 videos) — thanks to PR's like:

• PR Column error does not exist until lb dl is run #122
• PR Handle restricted/unavailable videos [and list them in "Tasks" view, when downloading (stale!) playlists] #179
• PR Ensure error and live_status columns are created from the start #191
• PR Adjust message to warn about problematic videos [e.g. "live" videos, that might not really be live anymore, but still fail to download] #194
• PR Adjust message for formerly "live" videos #211
• PR Tighten messaging of formerly live video's HH:MM:SS duration, in "Tasks" view (as it downloads) #216
• PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223

(2) editbooks.py: SAWarning: Object of type not in session, add operation along 'Authors.books' won't proceed

@deldesir suspects the above might be related to upstream Calibre-Web login sessions (authentication tokens, etc), and might have to be fixed upstream?