Merge pull request 'Merge: development into master' (#3) from develop…

…ment into master

Reviewed-on: https://gitea.catspeed.cc/catspeed-cc/tokenserver-debian/pulls/3

.gitmodules

@@ -0,0 +1,6 @@
+[submodule "submodules/--force"]
+	path = submodules/--force
+	url = https://gitea.catspeed.cc/catspeed-cc/youtube-trusted-session-generator.git
+[submodule "submodules/youtube-trusted-session-generator"]
+	path = submodules/youtube-trusted-session-generator
+	url = https://github.com/iv-org/youtube-trusted-session-generator

README.md

@@ -8,58 +8,58 @@ I am happy to announce github has restored my account :3c thank you GitHub for r
 
 Dockerized token server for catspeed fork found at https://github.com/catspeed-cc/invidious - based on Debian
 
-The project was basically finished in ~12 hours for the anonymous token pre-generator. Everything _was_ working, however something got buggered up and I could not find the problem. So I decided to move on to the token server and abandon the token pre-generator. You will still be pre-generating tokens though, because pulling them out of redis is the fastest way to get a response to the client.
+Tokenserver pre-generates and stores tokens in redis cache, allowing you to make requests to an API to get the tokens instantaneously. It is primarily designed to be used with catspeed-cc/invidious fork. Tokenserver can store many tokens and serve them randomly. Tokenserver utilizes https://github.com/catspeed-cc/youtube-trusted-session-generator a fork of https://github.com/iv-org/youtube-trusted-session-generator .
 
-I will also be moving the stats calculator here, to keep everything neat and tidy in this project, and also remove even more load off the invidious process/container. Eventually all that the invidious process/container will be doing is making redis calls.
+I will also be moving the stats calculator here, to keep everything neat and tidy in this project, and remove even more load off the invidious process/container. Eventually all that the invidious process/container will be doing is making redis calls.
 
-Invidious has enough troubles, serving content in a timely manner, and reliably as it is - after all the current recommendation is to restart it hourly, and I've even had to minutely just to stop it eating CPU and memory after the sig-helper crashes. Spawning cpu-intensive processes from within invidious when invidious is expected to make a timely response to a client is a bad idea. Invidious should be dedicated to serving users video in a timely manner. Even slight delays are unacceptable to the client.
+Tokenserver will be able to be set up behind a reverse proxy, and you will be able to have infinite number of token servers. The beautiful thing about this is the token data is only 350-500 bytes, meaning this can be set up on a relatively low bandwidth connection. I personally have only 10Mbit/sec upload, but it can handle lots at 500 bytes per request. Enough I suspect that it should sustain the catspeed invidious instance with high traffic. Worst case I can spin up a VPS with Vultr to take some load as well.
 
-So far in testing, the load on the invidious process/container has been massively reduced, because I no longer use the invidious main process to generate tokens (which was a bad idea anyways)
-
-Token server will be able to be set up behind a reverse proxy, and you will be able to have infinite number of token servers. The beautiful thing about this is the token data is only 350-500 bytes, meaning this can be set up on a relatively low bandwidth connection. I personally have only 10Mbit/sec upload, but it can handle lots at 500 bytes per request. Enough I suspect that it should sustain the catspeed invidious instance with high traffic. Worst case I can spin up a VPS with Vultr to take some load as well.
+Tokens must be generated from same VPN and/or IP address as the invidious instance (or other application needing tokens)
 
 Currently the api requires a trailing slash (ex. https://tokenserver.catspeed.cc/api/v1-00/get_tokens/) which is not a big deal, but I will try and fix this. Real API endpoint would not have a trailing slash. It's just some nginx configuration I have to work out.
 
-Turns out the tokens need to be generated from the same IP as the invidious server, and so a public token service will just not be possible.
-
 #### Token server will be compatible with other forks, as long as you know how to program in the API request and extraction of tokens from the JSON response.
 
 ## Features
 
+- Token generation
+- Redis cache
+- Token server / API
 - Stats monitor (not started yet)
-- Token generation (completed)
-- Token server / API (completed)
-- Catspeed integration (in testing)
 - Arm64 / Aarch64 image for raspi (not started yet)
+- Catspeed fork integration (in testing)
 
 ## Docker tags
 - catspeedcc/tokenserver-debian:latest - tag for latest version, can include minor version bumps (Ex. v0.50 -> v0.51)
 - catspeedcc/tokenserver-debian:stable - tag for stable version, only includes major version bumps (Ex. v1.00 -> v2.00 - COMING SOON!)
+- catspeedcc/tokenserver-debian:v0.54 - swap generators, additional docker service
+- catspeedcc/tokenserver-debian:v0.53 - revert back to submodules.
 - catspeedcc/tokenserver-debian:v0.52 - fixed gluetun - added sleep for gluetun init
 - catspeedcc/tokenserver-debian:v0.51 - fixed JSON output
 - catspeedcc/tokenserver-debian:v0.50 - initial image
 
 ## Releases
 
-- v0.52 is now released. You can find it on the releases/tags page. Includes gluetun fix.
+- v0.54 is now released. You can find it on the releases/tags page. Swapped YunzheZJU/youtube-po-token-generator for iv-org/youtube-trusted-session-generator. Added additional docker-compose service.
 
-The issue with gluetun was the git clone of the token generator was failing due to the VPN not being fully initialized. I have added a 30 second sleep which solves the issue. You will have to wait at least 30 seconds before tokens start to generate.
+I have added submodules back.
 
 **Note:** gluetun has been included in the example. Tokens must be generated from the same IP that you are accessing content from. In this case, we show example for gluetun because catspeed fork uses gluetun. You still have to select the proper server. If you are using catspeed fork, it has a better example in that repository.
 
 ## Dockerhub notes
 
-Even if you use the dockerhub image, you still require the git repository so you may as well clone it:
+Even if you use the dockerhub image, you still require the git repository:
 ```
 git clone https://github.com/catspeed-cc/tokenserver-debian
 ```
-This is due to the volumes linking to the token-data/ directory. Nothing I can do about it.
+This is due to the volumes linking to the token-data/ directory which you need to store data.
 
 ## Documentation
 
 #### Installation
 
 - ```git clone https://github.com/catspeed-cc/tokenserver-debian```
+- ```cd tokenserver-debian```
 - ```cp docker-compose.example.yml docker-compose.yml```
 - edit the docker-compose.yml file to your liking
 - ```docker-compose up -d```

docker-compose.example.yml

@@ -6,7 +6,7 @@ services:
     build:
       context: .
       dockerfile: docker/Dockerfile
-#    image: "catspeedcc/tokenserver-debian:latest"
+#    image: catspeedcc/tokenserver-debian:latest
     network_mode: "service:gluetun-1"
     init: true
     ulimits:
@@ -28,12 +28,21 @@ services:
       - SERVER_ID=catspeed1
       # IMPORTANT: enter the number of ANON tokens to generate 
       - NUM_TOKENS=10000
+      # IMPORTANT: enter the token expiry time 
+      - TOKEN_EXPIRY=1800
     volumes:
       - ./tokenserver-data/redis-data/:/var/lib/redis/
     secrets:
      - tokenserver-root-password
     depends_on:
       - gluetun-1
+      - session-generator-1
+
+  session-generator-1:
+    image: catspeedcc/youtube-trusted-session-generator:webserver-latest
+    network_mode: "service:gluetun-1"
+    depends_on:
+      - gluetun-1
 
   gluetun-1:
     image: ghcr.io/qdm12/gluetun
@@ -42,9 +51,11 @@ services:
     restart: unless-stopped
     devices:
       - /dev/net/tun:/dev/net/tun
-#    ports:
-#      - 8888:8888/tcp # HTTP proxy (expose to host is optional)
-#      - 8080:8080/tcp # Control port (expose to host is optional)
+    #ports:
+      #- 8880:80
+      #- 8881:8880
+      #- 8888:8888/tcp # HTTP proxy (expose to host is optional)
+      #- 8080:8080/tcp # Control port (expose to host is optional)
     volumes:
       - /docker/gluetun/data:/gluetun
     environment:

docker/Dockerfile

@@ -19,10 +19,12 @@ COPY ./etc/ssh/sshd_config /etc/ssh/sshd_config
 
 # upgrade & install dependencies
 RUN DEBIAN_FRONTEND=noninteractive \
-  apt-get update \
-  && apt-get upgrade -y \
-  && apt-get dist-upgrade -y \
-  && apt-get install -y htop nano git wget curl cpulimit bash net-tools nginx php8.2-fpm php8.2-redis redis-server openssh-server redis-tools nodejs npm postgresql-client
+  apt update ; \
+  apt upgrade -y ; \
+  apt dist-upgrade -y ; \
+  apt install -y htop nano git wget curl cpulimit net-tools nginx php8.2-fpm php8.2-redis \
+  redis-server openssh-server redis-tools nodejs npm postgresql-client python3 python3-full \
+  python3-virtualenv python3-pip ;
 
 # chown redis folder
 RUN chown -R redis:redis /var/lib/redis/

scripts/entrypoint.sh

@@ -43,49 +43,18 @@ chown -R www-data: /var/www/html/
 # SET ENVIRONMENT VARS
 echo "NUM_TOKENS=${NUM_TOKENS}" | tee -a /etc/environment
 echo "SERVER_ID=${SERVER_ID}" | tee -a /etc/environment
+echo "TOKEN_EXPIRY=${TOKEN_EXPIRY}" | tee -a /etc/environment
 
-# change to scripts/etc directory
-cd /scripts/
-
-# temporary test to see if entrypoint can git clone things
-ls -al | tee -a /scripts/entrypoint.log
-
-# make etc directory
-mkdir etc
-
-# change to scripts/etc directory
-cd /scripts/etc/
-
-# temporary test to see if entrypoint can git clone things
-ls -al | tee -a /scripts/entrypoint.log
-
-# git the REQUIRED token generator (using YunzheZJU until iv-org makes significant code changes, then will consider switch)
-git clone https://github.com/YunzheZJU/youtube-po-token-generator.git
+echo "Running curl cmd" | tee -a /scripts/entrypoint.log
 
-# git the catspeed projects (just do it :3c)
-git clone https://github.com/catspeed-cc/invidious.git
-git clone https://github.com/catspeed-cc/tokenserver-debian.git
-
-# temporary test to see if entrypoint can git clone things
-ls -al | tee -a /scripts/entrypoint.log
-
-# change to token generator directory
-cd /scripts/etc/youtube-po-token-generator/
-
-# install dependencies
-npm install
-
-# testrun the script
-echo "testing token generator" | tee -a /scripts/entrypoint.log
-node examples/one-shot.js | tee -a /scripts/entrypoint.log
+# localhost:8080 should be open now
+curl http://127.0.0.1:8880/token | tee -a /scripts/entrypoint.log
 
 # change back to scripts directory
 cd /scripts/
 
 # init token generation
-
-echo "starting token generation" | tee -a /scripts/entrypoint.log
-
+#echo "starting token generation" | tee -a /scripts/entrypoint.log
 /scripts/generate-tokens.sh &
 
 # this 'hack' will keep container awake and running

scripts/generate-tokens.sh

@@ -6,14 +6,17 @@ echo "" | tee -a /scripts/generate-tokens.log
 echo "generate-tokens.sh - for generating tokens - due to technical reasons, only works with catspeed fork!" | tee -a /scripts/generate-tokens.log
 echo "" | tee -a /scripts/generate-tokens.log
 
-echo "generating ${NUM_TOKENS} tokens" | tee -a /scripts/generate-tokens.log
+echo "generating ${NUM_TOKENS} tokens EXPY ${TOKEN_EXPIRY}" | tee -a /scripts/generate-tokens.log
 
 token_data=$(/usr/bin/node /scripts/etc/youtube-po-token-generator/examples/one-shot.js)
 echo "TEST token_data: '${token_data}'" | tee -a /scripts/generate-tokens.log
 
 REDIS_HOST=127.0.0.1
 REDIS_PORT=6379
 
+# to store previous tokens
+previous_token_data=
+
 # infinite loop because
 while true; 
 do
@@ -34,16 +37,31 @@ do
             echo "tokendata is empty, generating ..." | tee -a /scripts/generate-tokens.log
 
             # generate tokens :D
-
-            token_data=$(/usr/bin/node /scripts/etc/youtube-po-token-generator/examples/one-shot.js)
+            #token_data=$(/usr/bin/node /scripts/etc/youtube-po-token-generator/examples/one-shot.js)
+
+            # we are trying to generate tokens so we need to force update them
+            curl http://127.0.0.1:8880/update            
+
+            while [[ -z "${token_data}" ]] || [[ "${token_data}" == "${previous_token_data}" ]];
+            do
+
+              # loop until token not empty, and different from previous_token_data
+              token_data=$(curl http://127.0.0.1:8880/token) 
+
+              sleep 1           
+
+            done
+
+            # now we have new token, set previous so we can make sure next one is different
+            previous_token_data=$token_data
 
             token_data_len=${#token_data}
 
             if [[ $token_data_len -gt 25 ]]; then
 
-                redis-cli SET ${the_key} "${token_data}"
-                # temporarily removed "EX ${TOKEN_EXPIRY}"
+                redis-cli SET ${the_key} "${token_data}" EX ${TOKEN_EXPIRY}
                 echo "STORED IN REDIS: token_data: '${token_data}'" | tee -a /scripts/generate-tokens.log
+                echo "EXPY ${TOKEN_EXPIRY}" | tee -a /scripts/generate-tokens.log
 
             else
 

var/www/html/api/v1-00/get_tokens.php

@@ -31,63 +31,66 @@
 
 }
 
-// delete redis key, so all keys are unique - may as well :3c
-$redis->del($the_key);
+// pre-processing
+$token_data = str_replace("{", "{\n", $token_data);
+$token_data = str_replace("\"}", "\"\n}", $token_data);
+$token_data = str_replace(", ", ",\n", $token_data);
 
 // explode into an array for manipulation using \n
 $token_data_array = explode("\n", $token_data);
 
 // splice in element to the array at index 1
-$addtoarray="  \"server_id\": '$server_id',";
+$addtoarray="  \"server_id\": \"$server_id\",";
 array_splice($token_data_array, 1, 0, $addtoarray);
 
+// delete updated time
+array_splice($token_data_array, 2, 1);
+
 // swap element 2 & 3 because it bothers me
 $element2 = $token_data_array[2];
 $element3 = $token_data_array[3];
+//$element4 = $token_data_array[4];
 
 // ltrim element 2 & 3
 $element2 = ltrim($element2);
 $element3 = ltrim($element3);
+//$element4 = ltrim($element4);
 
 // explode each element
-$element2_array = explode(" ", $element2);
-$element3_array = explode(" ", $element3);
+//$element2_array = explode(" ", $element2);
+//$element3_array = explode(" ", $element3);
 
 // remove colon from elementX[0]
-$element2_array[0] = str_replace(":", "", $element2_array[0]);
-$element3_array[0] = str_replace(":", "", $element3_array[0]);
-
-// edit element2 2 & 3
-$element2_array[0] = "\"" . $element2_array[0] . "\":";
-$element3_array[0] = "\"" . $element3_array[0] . "\":";
+//$element2_array[0] = str_replace(":", "", $element2_array[0]);
+//$element3_array[0] = str_replace(":", "", $element3_array[0]);
 
 // implode element2 2 & 3
-$element2 = implode(" ", $element2_array);
-$element3 = implode(" ", $element3_array);
+//$element2 = implode(" ", $element2_array);
+//$element3 = implode(" ", $element3_array);
 
 // add left spaces back to element 2 & 3
 $element2 = "  " . $element2;
-$element3 = "  " . $element3;
+$element3 = "  " . $element3 . ",";
+//$element4 = "  " . $element4 . ",";
 
 // add comma to element3
-$element3 = $element3 . ",";
+//$element3 = $element3 . ",";
 
-// swap element 2 & 3
-$token_data_array[2] = $element3;
-$token_data_array[3] = $element2;
+// put elements back into array
+$token_data_array[2] = $element2;
+$token_data_array[3] = $element3;
+//$token_data_array[4] = $element4;
 
 // append error (OK) to end of array
 $theindex=max(array_keys($token_data_array));
-$addtoarray="  \"error\": 'OK'";
+$addtoarray="  \"error\": \"OK\"";
 array_splice($token_data_array, $theindex, 0, $addtoarray);
 
 // implode array back into string using \n
 $token_data = implode("\n", $token_data_array);
 
 // replace these because it bothers me
-$token_data = str_replace("poToken", "po_token", $token_data);
-$token_data = str_replace("visitorData", "visitor_data", $token_data);
-$token_data = str_replace("'", "\"", $token_data);
+$token_data = str_replace("potoken", "po_token", $token_data);
 
 // output the JSON formatted token data
 echo $token_data;