diff --git a/.breakpoints b/.breakpoints index aa67db3..791001a 100644 --- a/.breakpoints +++ b/.breakpoints @@ -1,5 +1,13 @@ { "files": { + "calibre-web-automater/setup-cwa.sh": [ + { + "id": "02c24718-34f4-4425-841f-dae976972cb5", + "line": 22, + "version": 278, + "index": 782 + } + ], "new-book-processor-test-version.py": [ { "id": "ee4a3fd4-14dc-4e16-b670-a7b76f57c1a0", diff --git a/.gitignore b/.gitignore index 6769e21..e42c93b 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,10 @@ __pycache__/ # C extensions *.so +# Test Scripts +test* +*dirs.json + # Distribution / packaging .Python build/ diff --git a/README.md b/README.md index ab292d2..2794df3 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,17 @@ After discovering that using the DOCKER_MODS universal-calibre environment varia - A **"fix"** that will address an issue many have with **Book Covers and Metadata** changed through the edit function of Calibre-Web, only changing visibly in Calibre-Web itself, not on your Kindle or other reading device - This new service / "fix" will come bundled with CWA's existing features and should be ready for release by Mid-July 2024 at the lastest +### New in Version 1.0.1: +- Fixed some pathing issues I missed in the release of Version 1.0.0 +- Simplified the Install process and updated the instructions here to match +- Added terminal commands you can use to change the given directories & check on the current status of the monitoring services +- Made preparations for Version 1.1.0 which is coming soon with new features + +### _Coming in Version 1.1.0:_ +- A "fix" that will address an issue many have with Book Covers and Metadata changed through the edit function of Calibre-Web, only changing visibly in Calibre-Web itself, not on your Kindle or other reading device + - This new service / "fix" will come bundled with CWA's existing features and should be ready for release by Mid-July 2024 at the lastest +- The ability to also use an **Calibre-Web-Automator** image from the DockerHub to install the service instead of having to modify an existing Calibre-Web Instance + ### ***Coming Soon :tm:*** - Ability to ***automatically push all newly imported books to your kindle*** through the existing **Send-to-Kindle** feature @@ -86,26 +97,31 @@ services: ~~~ ### CWA Installation ⚙️ + +1. Download the `calibre-web-automator` folder from this repo, unzip it, and then place the `calibre-web-automator` folder inside into the folder bound to your `/config` volume 2. Next, use the following command to gain access to the container's CLI, replacing ***calibre-web*** with the name of your Calibre-Web container if it differs: > `docker exec -it calibre-web bash` -3. Navigate inside the **CWA Install Folder** that you previously placed within your `/config` directory with the following command: - > `cd /config/calibre-web-automator` -4. Now initiate the install with the following command: +3. Navigate inside the **calibre-web-automator** that you previously placed within your `/config` directory with the following command: + > `cd /config/calibre-web-automator +4. Make sure the `setup-cwa.sh`is executable with the following command: + > `chmod +x setup-cwa.sh` +5. Now initiate the install with the following command: > `./setup-cwa.sh` -5. When prompted, follow the on-screen instructions to create and enter the paths of the directories the program needs to function. +6. When prompted, follow the on-screen instructions to create and enter the paths of the directories the program needs to function. - The folders can be wherever you like but **they must be in a persistent volume** like in your `/books` bind, **otherwise they and their contents won't be persistent between rebuilds of the container** -6. When the setup is complete, we need to restart the container for the changes to take effect. You can do so by using `exit` to return to your main shell and then running the following command: - > `docker restart calibre-web` -7. Once the container is back up and running, you should be good to go! To check however, do the following: - 1. Return to the CLI of the Calibre-Web container with `docker exec -it calibre-web bash` - 2. Navigate to the `calibre-web-automator` folder in your `/config` directory with `cd /config/calibre-web-automator` like before - 3. Then run the included testing script with `./check-cwa-install.sh` to verify your install. - - All three prompts should return green, indicating that the new `calibre-scan` and `books-to-process-scan` services are working properly. - - If one or both of the services return red indicating that they are not running, rebuild your Calibre-Web container using the `docker-compose` above and retry the installation process. +7. When the setup is complete, we need to restart the container for the changes to take effect. You can do so by using `exit` to return to your main shell and then running the following command: + > `docker restart calibre-web` or `docker restart ` +8. Once the container is back up and running, you should be good to go! To check however, do the following: + - Then run the included testing script with `cwa-check` anywhere in the terminal to verify your install. + - All three prompts should return green, indicating that the new `calibre-scan` and `books-to-process-scan` services are working properly. + - If one or both of the services return red indicating that they are not running, rebuild your Calibre-Web container using the `docker-compose` above and retry the installation process. -### Making The Changes persistent 🔗 +### Making The Changes Persistent 🔗 As you may know, everytime you rebuild a docker container, anything that isn't include in the source image or saved to a persistent volume, is gone and the container returns to it's stock state. @@ -116,7 +132,7 @@ As you may know, everytime you rebuild a docker container, anything that isn't i 1. Successfully install CWA using the steps above and confirm it's working by running the included `check-cwa-install.sh' binary from the CLI of your Calibre-Web container as described above in Step 7 2. While the container is running, from your main shell (use `exit` to return to your main shell if your still in the container's CLI) run the following command to generate an image of your newly modified Calibre-Web container, exactly as it's currently configured: > `docker commit calibre-web calibre-web-automated` - - Replace `calibre-web` with the name of your Calibre-Web container if it differs and you can also replace `calibre-web-automated` with whatever you like as it is only the name the image being generated will have + - Replace `calibre-web` with the name of your Calibre-Web container if it differs and you can also replace `calibre-web-automated` with whatever name you like 3. Once the process is finished, you can check the image was successfully created using the following command to list all current available docker images on your system: > `docker image ls` 4. Once you've confirmed the image was created successfully, edit your docker compose file so that the variable `image` is now as follows: @@ -133,7 +149,7 @@ services: 5. Finished! 🎉 Now everytime you rebuild your container, CWA as well as any other changes you may have made will remain 👍 ### Option 2: Re-Running 'setup-cwa.sh' Whenever You Rebuild the Container -This wouldn't be my preferred method but if you never really touch your containers the above may be overkill for you +This wouldn't be my preferred method but if you never really touch your containers the above may be overkill for you. How to Use 🔧 ----------- @@ -141,7 +157,7 @@ How to Use 🔧 1. Simply move your newly downloaded or existing eBook files to the ingest folder you designated during setup and anything you place in that folder will be automatically analysed, converted if necessary and then imported into your Calibre-Web library. 2. I personally use a script that my instance of qBittorrent will automatically execute upon finishing a download with the category **'books'** to fully automate the process however there's an infinite number of configurations out there so do whatever works best for your needs! 3. If you ever need to change the locations of your **ingest**, **import** and/ or **calibre-library** folders, you can do so in one of the following 2 ways: - 1. Edit the `dirs.json` file located in the `/etc/calibre-web-automator/` directory created during setup + 1. Use the 'cwa-change-dirs' command from anywhere in your terminal to open the json file where the paths are saved and change them as required. 2. Or simply reset your container to stock and rerun the setup script again, using the instructions above if needed Further Development 🏗️ diff --git a/calibre-web-automater/check-cwa-install.sh b/calibre-web-automater/check-cwa-install.sh deleted file mode 100644 index cbd7e56..0000000 --- a/calibre-web-automater/check-cwa-install.sh +++ /dev/null @@ -1,34 +0,0 @@ -RED='\033[0;31m' -GREEN='\033[0;32m' -NC='\033[0m' # No Color - -clear - -if s6-rc -a list | grep -q 'calibre-scan'; then - echo "" - echo -e "Calibre-scan ${GREEN}is running${NC}" - cs=true -else - echo "" - echo -e "Calibre-scan ${RED}is not running${NC}" - cs=false -fi - - -if s6-rc -a list | grep -q 'books-to-process-scan'; then - echo -e "Books-to-process-scan ${GREEN}is running${NC}" - bs=true -else - echo -e "Books-to-process-scan ${RED}is not running${NC}" - bs=false -fi - -if $cs && $bs; then - echo "" - echo -e "Calibre-Web-Automater was ${GREEN}sucsessfully installed ${NC}and ${GREEN}is running properly!${NC}" - echo "" -else - echo "" - echo -e "Calibre-Web-Automater was ${RED}not installed sucsessfully${NC}, please check the logs for more information." - echo "" -fi \ No newline at end of file diff --git a/calibre-web-automater/setup-cwa.sh b/calibre-web-automater/setup-cwa.sh deleted file mode 100644 index bd5b984..0000000 --- a/calibre-web-automater/setup-cwa.sh +++ /dev/null @@ -1,35 +0,0 @@ -#! /bin/bash - -# Script to automatically enable the automatic importing of epubs from the 'to_calibre' import folder upon container restart -# For help with S6 commands ect.: https://wiki.artixlinux.org/Main/S6 - -apt install -y inotify-tools -apt install -y python3 -apt install -y python3-pip - -# Loctation of this current script -SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" - -# Run setup.py to get dirs from user and store them in dirs.json -python3 $SCRIPT_DIR/setup.py - -# Copy book processing python script & dirs.json to it's own directory in /etc -mkdir /etc/calibre-web-automater -cp "$SCRIPT_DIR/new-book-processor.py" /etc/calibre-web-automater/new-book-processor.py -cp "$SCRIPT_DIR/dirs.json" /etc/calibre-web-automater/dirs.json - -mkdir /etc/s6-overlay/s6-rc.d/calibre-scan -echo "longrun" >| /etc/s6-overlay/s6-rc.d/calibre-scan/type -echo "bash run.sh" >| /etc/s6-overlay/s6-rc.d/calibre-scan/up -cp "$SCRIPT_DIR/calibre-scan.sh" /etc/s6-overlay/s6-rc.d/calibre-scan/run -touch /etc/s6-overlay/s6-rc.d/user/contents.d/calibre-scan - -mkdir /etc/s6-overlay/s6-rc.d/books-to-process-scan -echo "longrun" >| /etc/s6-overlay/s6-rc.d/books-to-process-scan/type -echo "bash run.sh" >| /etc/s6-overlay/s6-rc.d/books-to-process-scan/up -cp "$SCRIPT_DIR/books-to-process-scan.sh" /etc/s6-overlay/s6-rc.d/books-to-process-scan/run -touch /etc/s6-overlay/s6-rc.d/user/contents.d/books-to-process-scan - -echo " " -echo "SUCSESS: calibre-scan & books-to-process-scan setup complete, please restart the container so the changes will take effect" -echo "Then run 's6-rc -a list' to check if calibre-scan & books-to-process-scan are in the list of running services" \ No newline at end of file diff --git a/calibre-web-automater/books-to-process-scan.sh b/calibre-web-automator/books-to-process-scan.sh similarity index 89% rename from calibre-web-automater/books-to-process-scan.sh rename to calibre-web-automator/books-to-process-scan.sh index 9d24134..d6ecf24 100644 --- a/calibre-web-automater/books-to-process-scan.sh +++ b/calibre-web-automator/books-to-process-scan.sh @@ -8,13 +8,13 @@ echo "STARTING NEW-BOOK-PROCESSING SCANNER" apt install inotify-tools # Folder to monitor, replace "/books/to_process" with the folder you want to monitor e.g. your download folder for books -WATCH_FOLDER=grep -o '"ingest_folder": "[^"]*' /etc/calibre-web-automater/dirs.json | grep -o '[^"]*$' +WATCH_FOLDER=grep -o '"ingest_folder": "[^"]*' /etc/calibre-web-automator/dirs.json | grep -o '[^"]*$' echo "Watching folder: $WATCH_FOLDER" # Monitor the folder for new files inotifywait -m -e create -e moved_to "$WATCH_FOLDER" | while read -r directory events filename; do echo "PROCESSING: New files detected." - python3 /config/new-book-processor.py + python3 /etc/calibre-web-automator/new-book-processor.py echo "PROCESSING: New files sucsessfully moved/converted, the to_process folder has been emptied and is ready to go again." done \ No newline at end of file diff --git a/calibre-web-automater/calibre-scan.sh b/calibre-web-automator/calibre-scan.sh similarity index 93% rename from calibre-web-automater/calibre-scan.sh rename to calibre-web-automator/calibre-scan.sh index fc7371b..e9a28ef 100644 --- a/calibre-web-automater/calibre-scan.sh +++ b/calibre-web-automator/calibre-scan.sh @@ -8,11 +8,11 @@ echo "STARTING CALIBRE SCANNER" apt install inotify-tools # Folder to monitor -WATCH_FOLDER=grep -o '"import_folder": "[^"]*' /etc/calibre-web-automater/dirs.json | grep -o '[^"]*$' +WATCH_FOLDER=grep -o '"import_folder": "[^"]*' /etc/calibre-web-automator/dirs.json | grep -o '[^"]*$' echo "Watching folder: $WATCH_FOLDER" # Calibre library path -CALIBRE_LIBRARY=grep -o '"calibre_library_dir": "[^"]*' /etc/calibre-web-automater/dirs.json | grep -o '[^"]*$' +CALIBRE_LIBRARY=grep -o '"calibre_library_dir": "[^"]*' /etc/calibre-web-automator/dirs.json | grep -o '[^"]*$' echo "Calibre library: $CALIBRE_LIBRARY" # Function to add new eBook to Calibre database diff --git a/calibre-web-automator/check-cwa-install.sh b/calibre-web-automator/check-cwa-install.sh new file mode 100644 index 0000000..a7e9801 --- /dev/null +++ b/calibre-web-automator/check-cwa-install.sh @@ -0,0 +1,34 @@ +#!/bin/bash + +RED='\033[0;31m' +GREEN='\033[0;32m' +NC='\033[0m' # No Color + +# Print promt title +echo "==== Calibre-Web Automator -- Status of Monitoring Services ====" +echo "" + +if s6-rc -a list | grep -q 'calibre-scan'; then + echo "- Calibre-scan ${GREEN}is running${NC}" + cs=true +else + echo "- Calibre-scan ${RED}is not running${NC}" + cs=false +fi + + +if s6-rc -a list | grep -q 'books-to-process-scan'; then + echo "- Books-to-process-scan ${GREEN}is running${NC}" + bs=true +else + echo "- Books-to-process-scan ${RED}is not running${NC}" + bs=false +fi + +echo "" + +if $cs && $bs; then + echo "Calibre-Web-Automater was ${GREEN}sucsessfully installed ${NC}and ${GREEN}is running properly!${NC}" +else + echo "Calibre-Web-Automater was ${RED}not installed sucsessfully${NC}, please check the logs for more information." +fi \ No newline at end of file diff --git a/calibre-web-automater/new-book-processor.py b/calibre-web-automator/new-book-processor.py similarity index 98% rename from calibre-web-automater/new-book-processor.py rename to calibre-web-automator/new-book-processor.py index 0430756..408e95f 100644 --- a/calibre-web-automater/new-book-processor.py +++ b/calibre-web-automator/new-book-processor.py @@ -7,7 +7,7 @@ hierarchy_of_succsess = ['lit', 'mobi', 'azw', 'epub', 'azw3', 'fb2', 'fbz', 'azw4', 'prc', 'odt', 'lrf', 'pdb', 'cbz', 'pml', 'rb', 'cbr', 'cb7', 'cbc', 'chm', 'djvu', 'snb', 'tcr', 'pdf', 'docx', 'rtf', 'html', 'htmlz', 'txtz', 'txt'] dirs = {} -with open('dirs.json', 'r') as f: # '/etc/calibre-web-automater/dirs.json' +with open('/etc/calibre-web-automator/dirs.json', 'r') as f: dirs: dict[str, str] = json.load(f) # Both folders are assigned by user during setup diff --git a/calibre-web-automator/setup-cwa.sh b/calibre-web-automator/setup-cwa.sh new file mode 100644 index 0000000..f29bb39 --- /dev/null +++ b/calibre-web-automator/setup-cwa.sh @@ -0,0 +1,58 @@ +#!/bin/bash + +GREEN='\033[0;32m' +NC='\033[0m' # No Color + +# Script to automatically enable the automatic importing of epubs from the 'to_calibre' import folder upon container restart +# For help with S6 commands ect.: https://wiki.artixlinux.org/Main/S6 + +# Install required packages +apt install -y inotify-tools +apt install -y python3 +apt install -y python3-pip +apt install -y nano + +# Loctation of this current script +SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" + +# Make sure other sctipts are executable and permissions are correct +chown -R abc:users /config +chmod +x $SCRIPT_DIR/check-cwa-install.sh +chmod +x $SCRIPT_DIR/books-to-process-scan.sh +chmod +x $SCRIPT_DIR/calibre-scan.sh + +# Run setup.py to get dirs from user and store them in dirs.json +python3 $SCRIPT_DIR/setup.py + +# Copy book processing python script & dirs.json to it's own directory in /etc +mkdir /etc/calibre-web-automator +cp "$SCRIPT_DIR/new-book-processor.py" /etc/calibre-web-automator/new-book-processor.py +cp "$SCRIPT_DIR/dirs.json" /etc/calibre-web-automator/dirs.json +cp "$SCRIPT_DIR/check-cwa-install.sh" /etc/calibre-web-automator/check-cwa-install.sh + +# Add aliases to .bashrc +echo "" | cat >> ~/.bashrc +echo "# Calibre-Web Automator Aliases" | cat >> ~/.bashrc +echo "alias cwa-check='sh /config/check-cwa-install.sh'" | cat >> ~/.bashrc +echo "alias cwa-change-dirs='nano /etc/calibre-web-automater/dirs.json'" | cat >> ~/.bashrc +source ~/.bashrc + +# Setup inotify to watch for changes in the 'to_calibre' folder +mkdir /etc/s6-overlay/s6-rc.d/calibre-scan +echo "longrun" >| /etc/s6-overlay/s6-rc.d/calibre-scan/type +echo "bash run.sh" >| /etc/s6-overlay/s6-rc.d/calibre-scan/up +cp "$SCRIPT_DIR/calibre-scan.sh" /etc/s6-overlay/s6-rc.d/calibre-scan/run +touch /etc/s6-overlay/s6-rc.d/user/contents.d/calibre-scan + +# Setup inotify to watch for changes in the 'to_process' folder +mkdir /etc/s6-overlay/s6-rc.d/books-to-process-scan +echo "longrun" >| /etc/s6-overlay/s6-rc.d/books-to-process-scan/type +echo "bash run.sh" >| /etc/s6-overlay/s6-rc.d/books-to-process-scan/up +cp "$SCRIPT_DIR/books-to-process-scan.sh" /etc/s6-overlay/s6-rc.d/books-to-process-scan/run +touch /etc/s6-overlay/s6-rc.d/user/contents.d/books-to-process-scan + +# Setup completion notification +echo "${GREEN}SUCSESS${NC}: calibre-scan & books-to-process-scan setup complete!" +echo " - Please restart the container so the changes will take effect by typing 'exit' then presing enter, then running the docker command:" +echo " docker restart " +echo "\nTo check if the container is running properly followin the restart, use the command 'cwa-check' in the container's terminal." \ No newline at end of file diff --git a/calibre-web-automater/setup.py b/calibre-web-automator/setup.py similarity index 61% rename from calibre-web-automater/setup.py rename to calibre-web-automator/setup.py index 264bd96..dce2d2b 100644 --- a/calibre-web-automater/setup.py +++ b/calibre-web-automator/setup.py @@ -4,7 +4,6 @@ def main(): - while True: dirs = {} dirs |= {"ingest_folder":get_ingest_dir()} @@ -19,10 +18,14 @@ def main(): with open("dirs.json", "w") as f: json.dump(dirs, f) + os.system("chown abc:users dirs.json") sys.exit(1) def path_check(path: str) -> bool: - """Returns true if a given path exists and is a directory and False if not""" + """Returns true if a given path exists and is a directory after making sure it has the correct permissions, and False if not""" + if os.path.exists(f"{path}/"): + os.system(f'chown -R abc:1000 {path}/') + return os.path.exists(f"{path}/") def path_correct(path: str) -> str: @@ -38,13 +41,13 @@ def confirm_dirs(dirs: dict[str, str]) -> bool: """Allows the user to confirm that the entrered dirs are correct""" while True: os.system('cls' if os.name == 'nt' else 'clear') - print("\n-- Welcome to the Calibre-Web Automater setup script! --\n") + print("============ Welcome to the Calibre-Web Automater setup script! ============") - print("Please confirm the following directories are correct:\n") + print("\nPlease confirm the following directories are correct:\n") print(f"{'Ingest folder:':<30}{dirs['ingest_folder']}") print(f"{'Import folder:':<30}{dirs['import_folder']}") print(f"{'Calibre Library dir:':<30}{dirs['calibre_library_dir']}") - confirmation = input("\nAre these directories correct? (y/n): ").strip().lower() + confirmation = input("\nAre these directories correct? (Y/n): ").strip().lower() match confirmation: case "y": input("\nDirectories sucsessfully confirmed. Press Enter to continue the Setup.") @@ -59,11 +62,12 @@ def get_ingest_dir() -> str: """Gets the ingest folder from the user""" while True: os.system('cls' if os.name == 'nt' else 'clear') - print("\n-- Welcome to the Calibre-Web Automater setup script! --\n") + print("============ Welcome to the Calibre-Web Automater Setup Wizard! ============") - print("\nCalibre-Web-Automater needs an ingest folder for new files that are to be processed (converted / imported).") - print("This folder needs to be accessable from within the Calibre-Web container so make sure you add the appropriate binds to yopur docker compose file.\n") - ingest_folder = path_correct(input("Please make such a folder and enter it's internal container path here (e.g. /books/to_process/): ").strip()) + print("\nCalibre-Web-Automater needs an Ingest Folder for New Files that are to be\nprocessed (converted / imported if they're already epubs).\n") + print("This folder needs to be accessable from within the Calibre-Web container so\nmake sure you add the appropriate binds to your Docker-Compose file.\n") + print("Please make such a folder & enter it's internal container path below\n(e.g. /books/cwa-ingest/):\n") + ingest_folder = path_correct(input(" - Ingest Directory Path: ").strip()) if path_check(ingest_folder): return ingest_folder else: @@ -74,11 +78,12 @@ def get_import_dir() -> str: """Gets the import folder from the user""" while True: os.system('cls' if os.name == 'nt' else 'clear') - print("\n--Welcome to the Calibre-Web Automater setup script! --\n") + print("============ Welcome to the Calibre-Web Automater Setup Wizard! ============") - print("\nCalibre-Web-Automater also needs a folder for processed files to be temporarily stored within prior to their auto-import into Calibre-Web") - print("This folder also needs to be accessable from within the Calibre-Web container so make sure you add the appropriate binds to yopur docker compose file.\n") - import_folder = path_correct(input("Make such a folder & enter it's internal container path here (e.g. /books/to_calibre/): ").strip()) + print("\nCalibre-Web-Automater also needs a folder for processed files to be\ntemporarily stored within prior to their auto-import into Calibre-Web.\n") + print("This folder also needs to be accessable from within the Calibre-Web container\nso make sure you add the appropriate binds to yopur docker compose file.\n") + print("Make such a folder & enter it's internal container path below\n(e.g. /books/calibre-web_import/):\n") + import_folder = path_correct(input(" - Import Directory Path: ").strip()) if path_check(import_folder): return import_folder else: @@ -89,11 +94,12 @@ def get_calibre_library_dir() -> str: """Gets the path to the calibre library dir from the user""" while True: os.system('cls' if os.name == 'nt' else 'clear') - print("\n-- Welcome to the Calibre-Web Automater setup script! --\n") - - print("\nLastly, Calibre-Web-Automater needs the location of the calibre library folder in your container.") - print('It is usually "/calibre-main/Calibre Library".\n') - calibre_library_dir = path_correct(input("Please enter it's internal container path here: ").strip()) + print("============ Welcome to the Calibre-Web Automater Setup Wizard! ============") + + print("\nLastly, Calibre-Web-Automater needs the location of the Calibre Library\nfolder accessable from inside your container.\n") + print('It is usually "/calibre-main/Calibre Library/".\n') + print("Please enter it's internal container path below:\n") + calibre_library_dir = path_correct(input(" - Calibre Library Path: ").strip()) if path_check(calibre_library_dir): return calibre_library_dir else: