Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi OS w/ local_vars_unittest.yml — but installing Kolibri or Admin Console adds the missing dependency — which appears as "ModuleNotFoundError: No module named 'cryptography'" #260

Open
holta opened this issue Sep 26, 2024 · 39 comments
Assignees
Labels
bug Something isn't working question Further information is requested

Comments

@holta
Copy link
Member

holta commented Sep 26, 2024

Thanks @EMG70 for reporting!

Excerpt from his iiab-diagnostics (https://paste.centos.org/view/e76439c0) here:

=IIAB==========================================================================
COMMAND: /usr/bin/systemctl status calibre-web    # Is Calibre-Web running?
 
× calibre-web.service - Calibre-Web
     Loaded: loaded (/etc/systemd/system/calibre-web.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Thu 2024-09-26 15:28:47 BST; 3min 31s ago
   Duration: 5.596s
   Main PID: 575 (code=exited, status=1/FAILURE)
        CPU: 2.231s
 
Sep 26 15:28:47 box python3[575]:   File "/usr/local/calibre-web-py3/cps.py", line 28, in <module>
Sep 26 15:28:47 box python3[575]:     from cps.main import main
Sep 26 15:28:47 box python3[575]:   File "/usr/local/calibre-web-py3/cps/__init__.py", line 40, in <module>
Sep 26 15:28:47 box python3[575]:     from . import config_sql
Sep 26 15:28:47 box python3[575]:   File "/usr/local/calibre-web-py3/cps/config_sql.py", line 27, in <module>
Sep 26 15:28:47 box python3[575]:     from cryptography.fernet import Fernet
Sep 26 15:28:47 box python3[575]: ModuleNotFoundError: No module named 'cryptography'
Sep 26 15:28:47 box systemd[1]: calibre-web.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 15:28:47 box systemd[1]: calibre-web.service: Failed with result 'exit-code'.
Sep 26 15:28:47 box systemd[1]: calibre-web.service: Consumed 2.231s CPU time.
 
=IIAB==========================================================================
COMMAND: /usr/bin/journalctl -u calibre-web | tail -100    # Calibre-Web systemd log
 
Sep 26 15:19:00 box systemd[1]: Started calibre-web.service - Calibre-Web.
Sep 26 15:19:02 box python3[26014]: Traceback (most recent call last):
Sep 26 15:19:02 box python3[26014]:   File "/usr/local/calibre-web-py3/cps.py", line 28, in <module>
Sep 26 15:19:02 box python3[26014]:     from cps.main import main
Sep 26 15:19:02 box python3[26014]:   File "/usr/local/calibre-web-py3/cps/__init__.py", line 40, in <module>
Sep 26 15:19:02 box python3[26014]:     from . import config_sql
Sep 26 15:19:02 box python3[26014]:   File "/usr/local/calibre-web-py3/cps/config_sql.py", line 27, in <module>
Sep 26 15:19:02 box python3[26014]:     from cryptography.fernet import Fernet
Sep 26 15:19:02 box python3[26014]: ModuleNotFoundError: No module named 'cryptography'
Sep 26 15:19:03 box systemd[1]: calibre-web.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 15:19:03 box systemd[1]: calibre-web.service: Failed with result 'exit-code'.
Sep 26 15:19:03 box systemd[1]: calibre-web.service: Consumed 2.270s CPU time.
-- Boot 9dc10f09d20241c3990f05a440f96b5a --
Sep 26 15:23:27 box systemd[1]: Started calibre-web.service - Calibre-Web.
Sep 26 15:23:33 box python3[592]: Traceback (most recent call last):
Sep 26 15:23:33 box python3[592]:   File "/usr/local/calibre-web-py3/cps.py", line 28, in <module>
Sep 26 15:23:33 box python3[592]:     from cps.main import main
Sep 26 15:23:33 box python3[592]:   File "/usr/local/calibre-web-py3/cps/__init__.py", line 40, in <module>
Sep 26 15:23:33 box python3[592]:     from . import config_sql
Sep 26 15:23:33 box python3[592]:   File "/usr/local/calibre-web-py3/cps/config_sql.py", line 27, in <module>
Sep 26 15:23:33 box python3[592]:     from cryptography.fernet import Fernet
Sep 26 15:23:33 box python3[592]: ModuleNotFoundError: No module named 'cryptography'
Sep 26 15:23:33 box systemd[1]: calibre-web.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 15:23:33 box systemd[1]: calibre-web.service: Failed with result 'exit-code'.
Sep 26 15:23:33 box systemd[1]: calibre-web.service: Consumed 2.147s CPU time.
-- Boot b31bf14330474d1cbd2acf8461372eb0 --
Sep 26 15:28:41 box systemd[1]: Started calibre-web.service - Calibre-Web.
Sep 26 15:28:47 box python3[575]: Traceback (most recent call last):
Sep 26 15:28:47 box python3[575]:   File "/usr/local/calibre-web-py3/cps.py", line 28, in <module>
Sep 26 15:28:47 box python3[575]:     from cps.main import main
Sep 26 15:28:47 box python3[575]:   File "/usr/local/calibre-web-py3/cps/__init__.py", line 40, in <module>
Sep 26 15:28:47 box python3[575]:     from . import config_sql
Sep 26 15:28:47 box python3[575]:   File "/usr/local/calibre-web-py3/cps/config_sql.py", line 27, in <module>
Sep 26 15:28:47 box python3[575]:     from cryptography.fernet import Fernet
Sep 26 15:28:47 box python3[575]: ModuleNotFoundError: No module named 'cryptography'
Sep 26 15:28:47 box systemd[1]: calibre-web.service: Main process exited, code=exited, status=1/FAILURE
Sep 26 15:28:47 box systemd[1]: calibre-web.service: Failed with result 'exit-code'.
Sep 26 15:28:47 box systemd[1]: calibre-web.service: Consumed 2.231s CPU time.
@holta holta added the bug Something isn't working label Sep 26, 2024
@holta holta changed the title Calibre-Web installs but fails to work [502 bad gateway] with every Raspberry Pi OS installs — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] with every Raspberry Pi OS installs — ModuleNotFoundError: No module named 'cryptography' Sep 26, 2024
@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] with every Raspberry Pi OS installs — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] "on every recent Raspberry Pi OS install" — ModuleNotFoundError: No module named 'cryptography' Sep 26, 2024
@holta
Copy link
Member Author

holta commented Sep 26, 2024

Another excerpt from the above iiab-diagnostics, just for the record!

=IIAB==========================================================================
COMMAND: cd /usr/local/calibre-web-py3; sudo git log --graph --oneline --decorate | head -50    # Calibre-Web version
 
*   bf685667 (HEAD -> master, origin/master, origin/HEAD) Merge branch 'janeczku:master' into master
|\  
| * 1eb9cd44 Improve message on error cover downlaod
| * eb656348 Fix #3153 (In Firefox, Github OAuth login goes into infinite redirect loops)
| * 2f39e51e Fix for #3151 (Invalid amazon link for uk)
| * 884f7666 Integrate advocate and change netiface requirement
| *   c54e9fe5 Updated slovenian language
| |\  
| | * d82944d0 Updated Slovenian translation
| |/  
* | 2f7ae47e Merge branch 'janeczku:master' into master
|\|
| * 97764acb Added missing locale for datepicker in Slovenian language
| * 53f0647c New Slovenian translation
| * b5ee07eb Revert "Update Slovenian translation"
| * 5fe13f53 Update Slovenian translation
| * 89a1dcfb Fix for #3145 (improved contrast for epub reader headline)
| * 10d0542e Removed obsolete code
| *   111c93c3 Merge remote-tracking branch 'it/patch-1'
| |\  
| | * 2d7cf1e6 Update italian
| * | 1d9471b2 Possible fix for #3150 (Invalid cross-device link) without coping metadata
| * | 88cb58c2 Fix for load metadata from amazon
| |/  
* | 486b7bbc Merge branch 'janeczku:master' into master
|\|
| * 481237dc Fix for #3131 (Enable/Disable "Separate Book Files from Library" issue)
| * 5e9dfc07 Test update
* |   265bb088 Merge pull request #242 from holta/update_title_sort-TO-create_functions
|\ \  
| * | 0035e822 cps/editbooks.py: RENAME update_title_sort TO create_functions FOR PR #240
|/ /  
* |   16a1d69f Merge pull request #240 from holta/12-upstream-commits
|\ \  
| * | 8c3ed8e0 12 upstream commits, 29 files changes, 2 merge conflicts resolved (cps/editbooks.py cps/uploader.py)
|/| |
| |/  
| *   ce720f40 Shelf sort algorithm is now saved (#3003)
| |\  
| | * c92115fa Update save shelf sort algorithm
| * | a1f9f03f Fix for #3130 (only last e-mail address is saved when specifying multiple e-reader mail addresses in user settings)
| * | 6a504673 Merge branch 'Develop':
| |\|
| | * 839118e4 Bugfixes from testrun
| | * fd730f66 Bugfix generate thumbnails on gdrive
| | * 1be0ff96 Bugfix load metadata Bugfix xchange author and title Better handling of incompatible iso-639 module on python3.12
| | * 6717683a Improvements merge metadata upload format Music icon only visible once if more than one audio format available
| | * 0f9b5d08 Fix cover handling on upload formats
| | * 1eb77c9c Upload (multiple) book formats with progress and merge the corresponding metadata into the book
| | * d8d9d405 First step format upload with progress
 
=IIAB==========================================================================
COMMAND: /usr/bin/sudo lb --version    # xklb version
 
2.9.058
 
=IIAB==========================================================================
COMMAND: /usr/bin/sudo yt-dlp --version    # yt-dlp version
 
2024.08.06

@holta
Copy link
Member Author

holta commented Sep 26, 2024

@deldesir investigated, and FYI was unable to reproduce this during a fresh install of SMALL-sized IIAB on 64-bit Raspberry Pi OS Lite on RPi 5!

@holta
Copy link
Member Author

holta commented Sep 26, 2024

  • @EMG70 have you seen above that @deldesir is not able to reproduce your "502 Bad Gateway" on his RPi 5?

  • Can we try to dig deeper in coming days, to try to figure out what happened on your RPi 4, with another attempt?

  • Let's remove the "bug" label if this turns out not to be reproducible?

@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] "on every recent Raspberry Pi OS install" — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but failed to work [502 Bad Gateway] on recent Raspberry Pi OS install — ModuleNotFoundError: No module named 'cryptography' Sep 26, 2024
@holta holta added the question Further information is requested label Sep 26, 2024
@holta
Copy link
Member Author

holta commented Sep 27, 2024

@EMG70 did a size ZERO-sized (unittest) and '1' (SMALL-sized) IIAB install, both failed to access Calibre-Web.

All built on a RPi 400.

He/We are starting to ask if something is arising from an OS change (Raspberry Pi OS) affecting RPi 4 & RPi 400 especially?

@EMG70
Copy link

EMG70 commented Sep 27, 2024

New IIAB size ZERO installed on a PI400 with 64-bit Raspberry Pi Lite.Cannot access IIAB home page nor calibre-web via 192.168.../books (all return 502 Bad Gateway).
When recommended change is done iiab/iiab#3809 (reply in thread) i get 403 Forbidden error message
IIAB-DIAGNOSTICS - https://paste.centos.org/view/d64c9ccc

@holta
Copy link
Member Author

holta commented Sep 27, 2024

Cannot access IIAB home page

Please remember that IIAB home page is not supposed to work when you use ZERO-sized (local_vars_unittest.yml) which contains:

admin_console_install: False
admin_console_enabled: False

https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/vars/local_vars_unittest.yml#L33-L34

@EMG70
Copy link

EMG70 commented Sep 27, 2024

Cannot access IIAB home page

Please remember that IIAB home page is not supposed to work when you use ZERO-sized (local_vars_unittest.yml) which contains:

admin_console_install: False
admin_console_enabled: False

https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/vars/local_vars_unittest.yml#L33-L34

My apologies.

@holta holta changed the title Calibre-Web installs but failed to work [502 Bad Gateway] on recent Raspberry Pi OS install — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi 4 and 400 — ModuleNotFoundError: No module named 'cryptography' Sep 27, 2024
@holta
Copy link
Member Author

holta commented Sep 27, 2024

@deldesir please focus on fixing this, as it appears to be affecting every RPi 4, every RPi 400 (and very possibly others too!)

@deldesir
Copy link
Collaborator

Hey @EMG70 and @holta , I will try to reproduce the issue on both RPI 4 and RPI 400. Will let you know...

@deldesir
Copy link
Collaborator

deldesir commented Sep 28, 2024

I tried to reproduce the bug on RPI4 (for now) without success:

Test 1: Reinstalling Calibre-Web on RPI 4 using runrole --reinstall calibre-web passed: https://paste.centos.org/view/185ad7e4
Test 2: Reinstalling Calibre-Web on RPI 4 using runrole --reinstall calibre-web. This time the /usr/local/calibre-wep-py3 file was deleted prior running the command. https://paste.centos.org/view/13072dff
Test 3: Fresh install with Raspberry Pi OS Lite (64bit) on RPI 4: Could not proceed fully as iiab.io is not reachable for now. I used a copy of the install script but I don't have the scripts/binaries hosted at iiab.io at hand https://paste.centos.org/view/500ccacf

@holta
Copy link
Member Author

holta commented Sep 28, 2024

You don't need binaries from iiab.io during unit testing:

  1. Start with https://raw.githubusercontent.com/iiab/iiab/refs/heads/master/vars/local_vars_unittest.yml (a.k.a. ZERO-sized) as your /etc/iiab/local_vars.yml to keep focus on just 1 app.

  2. Also set these in your /etc/iiab/local_vars.yml

    usb_lib_install: False
    usb_lib_enabled: False
    
  3. Change Line 99 of /opt/iiab/iiab/roles/www_options/tasks/main.yml to just about any working URL — instead of https://download.iiab.io/packages/heart-beat.txt

    In other words Line 99 here:
    https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/roles/www_options/tasks/main.yml#L99

@holta
Copy link
Member Author

holta commented Sep 28, 2024

3. Change Line 99 of /opt/iiab/iiab/roles/www_options/tasks/main.yml to just about any working URL — instead of https://download.iiab.io/packages/heart-beat.txt
In other words Line 99 here:
https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/roles/www_options/tasks/main.yml#L99

Better yet, completely comment out these 20 lines — i.e. the scraping of IIAB Docs:

https://github.com/iiab/iiab/blob/4dd927cfd00c3bed1a36b9c197f554d7ee680ceb/roles/www_options/tasks/main.yml#L96-L115

(e.g. in situations where FAQ.IIAB.IO and IIAB.IO are both unreachable.)

PS easiest way to do that is to set... nodocs: True

@holta
Copy link
Member Author

holta commented Sep 28, 2024

  1. During rapid unit-testing, you almost certainly also want:
network_install: False
network_enabled: False

@deldesir
Copy link
Collaborator

Issue reproduced on a fresh install on RPI 4:
https://paste.centos.org/view/f177f711

@holta
Copy link
Member Author

holta commented Sep 28, 2024

@deldesir suggests the "gone missing" cryptography module is very likely related to... piwheels or Raspberry Pi OS or similar... with these 2023 discussions being extremely relevant:

@holta
Copy link
Member Author

holta commented Sep 28, 2024

  1. @deldesir does a modern/appropriate version of running...

    pip install cryptography
    

    ...move the ball forward during testing?

  2. Is the release of cryptography 43.0.1 on Sep 3, 2024 relevant?

    https://pypi.org/project/cryptography/#history

@holta
Copy link
Member Author

holta commented Sep 28, 2024

2. Is the release of cryptography 43.0.1 on Sep 3, 2024 relevant?

https://pypi.org/project/cryptography/#history

Or cryptography 43.0.0 from 2024-07-20 ?

Screenshot_20240928-163558

SEE: https://www.piwheels.org/project/cryptography/

Can you please manually try cryptography 42.0.8 to see if that helps? If so, reverting to 42.0.8 should be easy to hard-code later, as a temporary patch / workaround revising Calibre-web's requirements.txt file upstream... or here if necessary:

@holta
Copy link
Member Author

holta commented Sep 28, 2024

2023 discussions being extremely relevant:

@deldesir is this possibly the root cause?

@holta
Copy link
Member Author

holta commented Sep 29, 2024

@deldesir to avoid broken Calibre-Web installs like this in future, please document: (as soon as you can, here on this ticket!)

  • Version of Python cryptography (docs) that's now in use and working on Raspberry Pi 5?
  • What's likely failing to install on Raspberry Pi 4 and 400 and why?
    • Most immediate path to get Calibre-Web working again on Raspberry Pi 4 and 400?
  • What dependencies / versions likewise need to be validated on Raspberry Pi Zero 2 W and Raspberry Pi 3?
    • How? e.g. so everyone can check their own machine (command can be a part of iiab-diagnostics if necessary?!)

➡️ If an ugly/urgent workaround patch is necessary in the short term that's fine — but a proper analysis of what went wrong is even more important — to make very clear how we all avoid such situations in future... ! 🦅

@EMG70
Copy link

EMG70 commented Sep 29, 2024

I have just installed IIAB size 1 ON A PI 4 and calibre-web is reachable.I ran sudo apt update && sudo apt -y full-upgrade before installing IIAB
SUDO IIAB-DIAGNOSTICS https://paste.centos.org/view/8d283b47
pastebinit -b paste.centos.org /var/log/apt/history.log https://paste.centos.org/view/5eca0560

@holta
Copy link
Member Author

holta commented Sep 29, 2024

Thanks @EMG70 !

@deldesir should look inside your /var/log/apt/history.log (https://paste.centos.org/view/5eca0560) to investigate whether any very recent apt update fix[es] (e.g. python3-netifaces=0.11.0-2+b1 ? or similar?) were possibly just pushed by Raspberry Pi OS for RPi 4 & 400, in the last day or two?

@holta
Copy link
Member Author

holta commented Sep 29, 2024

apt package python3-netifaces appears irrelevant to the problem. It's unchanged since almost 2 years:

root@box:~# apt list python3-netifaces
Listing... Done
python3-netifaces/stable,now 0.11.0-2+b1 arm64 [installed]
python3-netifaces/stable 0.11.0-2+b1 armhf
root@box:~# apt changelog python3-netifaces

netifaces (0.11.0-2) unstable; urgency=medium

  [ Debian Janitor ]
  * Bump debhelper from old 12 to 13.
  * Update standards version to 4.6.1, no changes needed.

 -- Jelmer Vernooij <[email protected]>  Wed, 30 Nov 2022 17:06:23 +0000

...

@deldesir (and others?) should re-test Raspberry Pi 400: try to figure out what's happening and why.

@holta
Copy link
Member Author

holta commented Sep 29, 2024

@deldesir please investigate @EMG70's very promising new theory — if this is confirmed, it should be easy for you to bisect & help all solve + fully explain the problem:

  • Problem happens with UNITTEST-sized installs of Calibre-Web
  • Problem doesn't happen with SMALL-sized & similar/larger /etc/iiab/local_vars.yml files

PS Does this mean that the subject line of this ticket is false, and the problem occurs on RPi 5 etc too?

@deldesir
Copy link
Collaborator

Interesting. I am going to test with small-size.

@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi 4 and 400 — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] w/ local_vars_unittest.yml on Raspberry Pi BUT works with local_vars_small.yml [?!] — ModuleNotFoundError: No module named 'cryptography' Sep 29, 2024
@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] w/ local_vars_unittest.yml on Raspberry Pi BUT works with local_vars_small.yml [?!] — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] w/ local_vars_unittest.yml on Raspberry Pi BUT works w/ local_vars_small.yml [?!] — ModuleNotFoundError: No module named 'cryptography' Sep 29, 2024
@holta
Copy link
Member Author

holta commented Sep 29, 2024

A new theory is obviously needed:

  • @EMG70 installed an IIAB UNITTEST (in other words, Calibre-Web only) onto RPi 400 and it worked perfectly: https://paste.centos.org/view/922ad244

  • Please all post details (iiab-diagnostics especially) from tests, no matter the result! Onto this ticket (260) to help solve this collaboratively + quickly.

  • Fiddling with setting network_install: False and network_enabled: False is probably complicating things unnecessarily. (But if absolutely necessary, that can be tried later.)

RECAP: Keep focus on completely clean/fresh "as minimal as possible" IIAB + Calibre-Web installs — onto RPi 4, RPi 400 & RPi 5 (and VM's if necessary, for comparison) — until the hidden cause/effect pattern finally becomes obvious! 🙏

@holta
Copy link
Member Author

holta commented Sep 29, 2024

Paranoid, but... maybe try to use the exact same hardware (exact same Raspberry Pi) for subsequent/repeated tests? e.g. use the very same RPi 4 unit for every install test, or the very same RPi 400 unit (so that comparisons between many tests... can be more meaningful?)

Now I realize this is absolute paranoia (!) but yeah, do eliminate as much ambiguity as possible... testing cause/effect ~ repeatability/reproducibility in the narrowest way possible... trying to be as scientific as possible... ! 🤔

@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] w/ local_vars_unittest.yml on Raspberry Pi BUT works w/ local_vars_small.yml [?!] — ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but often fails to work [502 Bad Gateway] on Raspberry Pi 4 & 400 especially? ModuleNotFoundError: No module named 'cryptography' Sep 29, 2024
@holta holta changed the title Calibre-Web installs but often fails to work [502 Bad Gateway] on Raspberry Pi 4 & 400 especially? ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi w/ local_vars_unittest.yml — but installing Admin Console adds the missing dependency: ModuleNotFoundError: No module named 'cryptography' Sep 29, 2024
@holta
Copy link
Member Author

holta commented Sep 29, 2024

A new theory is obviously needed:

Oops I misunderstood that @EMG70 had also enabled Admin Console in the above RPi 400 test ~1.5 hours ago. In other words, it wasn't a single unit test of Calibre-Web alone 😄 ...but coincidentally is very revealing as a result!

CONCLUSION / NEW THEORY: Installing Admin Console appears to install software that (very coincidentally) allows Calibre-Web to work.

@deldesir this should hopefully be very easy to solve now! Please when you can:

  1. Trace which exact software package this is... introduced during install of Admin Console... that obviously installs the cryptography module needed by Python (anything else we need!?) and...
  2. Explain (in detail, to protect everyone in future with different platforms / different hardware / etc) why this software package is apparently needed on some-or-all Raspberry Pis?

@deldesir
Copy link
Collaborator

deldesir commented Sep 29, 2024

Small-size install was successful: https://paste.centos.org/view/3ad4a9fb.
Thanks Ed for pointing this out.

@holta
Copy link
Member Author

holta commented Sep 29, 2024

@deldesir for completeness, can you plz confirm that a UNITTEST install (i.e. Calibre-Web alone) fails on RPi 5?

@deldesir
Copy link
Collaborator

Good idea. Will let you know.

@deldesir
Copy link
Collaborator

deldesir commented Sep 29, 2024

Ansible-core installs cryptography-43.0.1. It works with python 3.12 (ubuntu 24.04) but not with python 3.11 (RPI OS which is based on debian bookworm).
Calibre-Web works with small-size install on RPIs because kolibri installs python3-cryptography (version 38.0.4-3).

@deldesir
Copy link
Collaborator

deldesir commented Sep 29, 2024

As expected, Calibre-Web fails with a 404 error on RPI 5 using UNNITTEST install.
https://paste.centos.org/view/f71c375d

@holta
Copy link
Member Author

holta commented Sep 30, 2024

Ansible-core installs cryptography-43.0.1.

While interesting, isn't this off topic when IIAB installs ansible-core into its own venv?

Calibre-Web works with small-size install on RPIs because kolibri installs python3-cryptography (version 38.0.4-3).

Ok!

What precise package (and what precise line) of Admin Console's install adds the dependenc(ies) that allow Calibre-Web to work?

@holta
Copy link
Member Author

holta commented Sep 30, 2024

As expected, Calibre-Web fails with a 404 error on RPI 5 using UNNITTEST install.
https://paste.centos.org/view/f71c375d

Why 404?

(So far @EMG70 has only mentioned "502 Bad Gateway")

@holta
Copy link
Member Author

holta commented Sep 30, 2024

In conclusion:

  • What is the safest and lowest-maintenance way to fix roles/calibre-web/tasks/install.yml ?

  • Is using Ansible to install apt package python3-cryptography regrettably best in the end?

(Or, if there's an even lower-risk approach inside venv... that'd be even better... but perhaps that's a fantasy given recent versions of Python cryptography are all too often unavailable on Raspberry Pi OS!?)

@holta
Copy link
Member Author

holta commented Sep 30, 2024

As expected, Calibre-Web fails with a 404 error on RPI 5 using UNNITTEST install. https://paste.centos.org/view/f71c375d

As @EMG70 asks:

@deldesir don't you really want /etc/iiab/local_vars.yml to contain... ?

calibreweb_install: True
calibreweb_enabled: True

So that you end up with /etc/iiab/iiab.ini containing... ?

calibreweb_install = True
calibreweb_enabled = True

@holta
Copy link
Member Author

holta commented Sep 30, 2024

Official (upstream) Calibre-Web install instructions:

@holta
Copy link
Member Author

holta commented Sep 30, 2024

FYI even the most minimal Ubuntu and Debian OS images include the apt package python3-cryptography out-of-the-box:

root@u2410:~# aptitude why python3-cryptography
i   cloud-init       Depends python3-oauthlib
i A python3-oauthlib Depends python3-cryptography
root@deb12:~# aptitude why python3-cryptography
i   cloud-init       Depends python3-oauthlib
i A python3-oauthlib Depends python3-cryptography
root@u2204:~# aptitude why python3-cryptography
i   cloud-init       Depends python3-oauthlib
i A python3-oauthlib Depends python3-cryptography

Whereas this is apparently not true on Raspberry Pi OS... ?

@holta
Copy link
Member Author

holta commented Oct 1, 2024

  1. To figure out why Calibre-Web unit testing is not yet possible on Raspberry Pi OS, @deldesir made progress with me over the past 2 hours:

    • Running apt install python3-cryptography and apt install python3-pip is insufficient: Calibre-Web still does not work 😢
    • Installing Kolibri "fixes" a Calibre-Web installation — but we do not yet know why — as Raspberry Pi OS's /var/log/dpkg.log and /var/log/apt/history.log don't seem to show much! ❓
    • To try to figure out what's really happening, he (@deldesir) will try to bisect the install of ~10 apt packages by Admin Console, just in case one-or-more of these is the answer?! 🕵️ 🔎
      • python3-requests
      • python3-jinja2
      • python3-systemd
      • python3-zmq
      • python3-yaml
      • python3-cracklib
      • python3-nacl
      • python3-bs4
      • python3-lxml
      • uuid
  2. @deldesir please prepare a DRAFT (or well-tested!) PR for @EMG70 and others as soon as you solve this, so that rapid unit-testing of Calibre-Web becomes possible on Raspberry Pi OS at last, Thanks !

  3. Should we also ask Ozzie Isaacs (i.e. the maintainer of the upstream version of Calibre-Web) to make apt install package requirements more precise — e.g. in the official instructions at https://github.com/janeczku/calibre-web/wiki/Manual-installation#manual-installation-on-linux-or-macos ?

  4. @deldesir please try hard to explain (if you can!) why @EMG70 sees "502 Bad Gateway" whereas you see 404 — when attempting unit test installs of Calibre-Web on Raspberry Pi OS — and then visiting http://box/books (thank you !!)

@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi w/ local_vars_unittest.yml — but installing Admin Console adds the missing dependency: ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi OS w/ local_vars_unittest.yml — but installing Admin Console adds the missing dependency: ModuleNotFoundError: No module named 'cryptography' Oct 1, 2024
@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi OS w/ local_vars_unittest.yml — but installing Admin Console adds the missing dependency: ModuleNotFoundError: No module named 'cryptography' Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi OS w/ local_vars_unittest.yml — but installing Kolibri or Admin Console adds the missing dependenc(ies) — one of which appears as "ModuleNotFoundError: No module named 'cryptography'" Oct 1, 2024
holta added a commit to deldesir/iiab that referenced this issue Oct 1, 2024
@holta holta changed the title Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi OS w/ local_vars_unittest.yml — but installing Kolibri or Admin Console adds the missing dependenc(ies) — one of which appears as "ModuleNotFoundError: No module named 'cryptography'" Calibre-Web installs but fails to work [502 Bad Gateway] on Raspberry Pi OS w/ local_vars_unittest.yml — but installing Kolibri or Admin Console adds the missing dependency — which appears as "ModuleNotFoundError: No module named 'cryptography'" Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants