GitBook: [#9] No subject

docs/README.md

@@ -0,0 +1,2 @@
+# Explanations
+

docs/SUMMARY.md

@@ -0,0 +1,14 @@
+# Table of contents
+
+* [Explanations](README.md)
+  * [Docker Setup of pro platform for development](../explanations/pro-dev-setup.md)
+* [How to guides](../how-to-guides/README.md)
+  * [Prod environment deployment guide](../how-to-guides/deploy.md)
+  * [Docker Developer's Guide](../how-to-guides/docker-developer-guide.md)
+  * [Developing on the producers platform](../how-to-guides/pro-development.md)
+  * [Use direnv](../how-to-guides/use-direnv.md)
+  * [Using Gitpod for Remote Development](../how-to-guides/use-gitpod.md)
+  * [Using VSCode](../how-to-guides/use-vscode.md)
+  * [Using pages from openfoodfacts-web](../how-to-guides/using-pages-from-openfoodfacts-web.md)
+* [Introduction](../introduction/README.md)
+  * [Dev environment quick start guide](../introduction/dev-environment-quick-start-guide.md)

explanations/pro-dev-setup.md

@@ -0,0 +1,27 @@
+# Docker Setup of pro platform for development
+
+This explains how we setup docker file for pro platform development. For explanations on how to use it, see: [how-to-guides/pro-development](../how-to-guides/pro-development.md)
+
+OFF is the public facing application (world.openfoodfacts.org) off-pro is the producers platform (world.pro.openfoodfacts.org)
+
+When we work on the pro platform for development we want:
+
+* off containers to talk between each other, and have their own volumes
+* off-pro containers to talk between each other, and, generally, have their own volumes
+* minion and backend from both app access to the same postgres database (which stores tasks queues)
+* off and off-pro backends / minion needs to share some volumes : orgs, users ands some files living in podata
+
+Still we would like to avoid having different clone of the repository, but we can isolate projects thanks to `COMPOSE_PROJECT_NAME`, which will prefix containers names, volumes and default network, thus isolate each projects.
+
+This is achieved by sourcing the .env-pro which setup some environment variables that will superseed the .env variables. The main one being setting `COMPOSE_PROJECT_NAME` and `PRODUCERS_PLATFORM`, but also other like `MINION_QUEUE`.
+
+On volume side, we will simply give hard-coded names to volumes that should be shared between off and pro platform, thus they will be shared. Ideally we should not have to share single files but this is a work in progress, we will live without it as a first approx.
+
+To satisfy the access to the same database, we will use postgres database from off as the common database.
+
+In order to achieve that:
+
+* we use profiles, so we won't start postgres in pro docker-compose
+* we connect `postgres`, `backend` and `minion` services to a shared network, called `minion_db` Fortunately this works, but note that there is a pitfall: on `minion_db` network both `backend` services (`off` and `off-pro`) will respond to same name. For the moment it is not a problem for we don't need to communicate directly between instances. If it was, we would have to define custom aliases for those services on the `minion_db` network.
+
+network OFF network PRO network po\_default containers minion\_db containers po\_pro\_default | | | +------postgres------------+ | | | | | | | +-----backend--------------+ | | +----------backend-------------+ | | | +------minion--------------+ | | +----------minion--------------+ | | | | | | +------frontend | frontend------------+ +------mongodb | mongodb-------------+ | | |

how-to-guides/README.md

@@ -0,0 +1,2 @@
+# How to guides
+

how-to-guides/deploy.md

@@ -0,0 +1,22 @@
+# Prod environment deployment guide
+
+Note: prod deployment is very manual and not automated yet.
+
+* Login to the off1 server, as the "off" user
+* cd /home/off/openfoodfacts-server
+* Check that you are on the main branch
+* git pull
+* Copy changed files (don't copy everything, in particular not the lang directory that is being moved to the openfoodfacts-web repository)
+* e.g. cp cgi scripts lib po taxonomies templates /srv/off/
+* cd /srv/off
+* export NPM\_CONFIG\_PREFIX=\~/.npm-global
+* npm install
+* npm run build
+* cd /srv/off/cgi
+* export PERL5LIB=.
+* ./build\_lang.pl
+* as the root user:
+* systemctl stop apache2@off
+* systemctl start apache2@off
+* systemctl stop minion-off
+* systemctl start minion-off

how-to-guides/docker-developer-guide.md

@@ -0,0 +1,234 @@
+# Docker Developer's Guide
+
+This guide is for developers and newcomers to help them debug and explore Docker.
+
+This page describes how to test and debug your changes once you have set up the project, Product Opener with Docker using [dev environment quick start guide](../introduction/dev-environment-quick-start-guide.md).
+
+## Checking logs
+
+### Tail Docker Compose logs
+
+```
+make log
+```
+
+### Tail other logs
+
+```
+make tail
+```
+
+It will `tail -f` all the files present in the `logs/` directory:
+
+* `apache2/error.log`
+* `apache2/log4perl.log`
+* `apache2/modperl_error.log`
+* `apache2/other_vhosts_access.log`
+* `nginx/access.log`
+* `nginx/error.log`
+
+You can also simply run:
+
+```
+tail -f <FILEPATH>
+```
+
+to check a specific log.
+
+### Increasing log verbosity
+
+By default, the `log4perl` configuration `conf/log.conf` matches production settings. You can tweak that file with your own dev configuration settings and run `make restart` to reload the changes.
+
+A setting useful for local environments is to set `TRACE` log level:
+
+```
+log4perl.rootLogger=TRACE, LOGFILE
+```
+
+## Opening a shell in a Docker container
+
+Run the following to open a bash shell within the `backend` container:
+
+```
+docker-compose exec backend bash
+```
+
+You should see `root@<CONTAINER_ID>:/#` (opened root shell): you are now within the Docker container and can begin typing some commands !
+
+### Checking permissions
+
+Navigate to the directory the specific directory and run
+
+```
+ls -lrt
+```
+
+It will list all the directories and their permission status.
+
+### Creating directory
+
+Navigate to your specific directory using `cd` command and run
+
+```
+mkdir directory-name
+```
+
+### Running minion jobs
+
+[Minion](https://docs.mojolicious.org/Minion) is a high-performance job queue for Perl. [Minion](https://docs.mojolicious.org/Minion) is used in [openfoodfacts-server](https://github.com/openfoodfacts/openfoodfacts-server) for time-consuming import and export tasks. These tasks are processed and queued using the minion jobs queue. Therefore, they are called minion jobs.
+
+Go to `/opt/product-opener/scripts` and run
+
+```
+./minion_producers.pl minion job
+```
+
+The above command will show the status of minion jobs. Run the following command to launch the minion jobs.
+
+```
+./minion_producers.pl minion worker -m production -q pro.openfoodfacts.org
+```
+
+### Restarting Apache
+
+Sometimes restarting the whole `backend` container is overkill, and you can just restart `Apache` from inside the container:
+
+```
+apache2ctl -k restart
+```
+
+### Exiting the container
+
+Use `exit` to exit the container.
+
+## Making code changes
+
+In the dev environment, any code change to the local directory will be written on the container. That said, some code changes require a restart of the `backend` container, or rebuilding the NPM assets.
+
+### Manual reload
+
+To restart the `backend` container after a config (`docker-compose.yml`, `docker/`, `.env`):
+
+```
+make restart
+```
+
+To restart `Apache` within the backend container after a code change (`lib/`):
+
+```
+make restart_apache
+```
+
+**Note:** restart is not necessary when making changes in the `cgi/` directory.
+
+To rebuild frontend assets after an asset change (`html/` folder):
+
+```
+make up
+```
+
+### Live reload
+
+To automate the live reload on code changes, you can install the Python package `when-changed`:
+
+```
+pip3 install when-changed
+when-changed -r docker/ docker-compose.yml .env -c "make restart"                                         # restart backend container on compose changes
+when-changed -r lib/ -r docker/ docker-compose.yml -c "make restart_apache"                               # restart Apache on code changes
+when-changed -r html/ -r css/ -r scss/ -r icons/ Dockerfile Dockerfile.frontend package.json -c "make up" # rebuild containers on asset or Dockerfile changes
+```
+
+An alternative to `when-changed` is `inotifywait`.
+
+## Run queries on MongoDB database
+
+```
+docker-compose exec mongodb mongo
+```
+
+The above command will open a MongoDB shell, allowing you to use all the `mongo` commands to interact with the database:
+
+```
+show dbs
+use off
+db.products.count()
+db.products.find({_id: "5053990155354"})
+db.products.deleteOne({_id: "5053990155354"})
+```
+
+See the [`mongo` shell docs](https://docs.mongodb.com/manual/reference/mongo-shell/) for more commands.
+
+## Adding environment variables
+
+If you need some value to be configurable, it is best to set is as an environment variable.
+
+To add a new environment variable `TEST`:
+
+* In `.env` file, add `TEST=test_val` \[local].
+* In `.github/workflows/container-deploy.yml`, add `echo "TEST=${{ secrets.TEST }}" >> .env` to the "Set environment variables" build step \[remote]. Also add the corresponding GitHub secret `TEST=test_val`.
+* In `docker-compose.yml` file, add it under the `backend` > `environment` section.
+* In `conf/apache.conf` file, add `PerlPassEnv TEST`.
+* In `lib/Config2.pm`, add `$test = $ENV{TEST};`. Also add `$test` to the `EXPORT_OK` list at the top of the file to avoid a compilation error.
+
+The call stack goes like this:
+
+`make up` > `docker-compose` > loads `.env` > pass env variables to the `backend` container > pass to `mod_perl` > initialized in `Config2.pm`.
+
+## Managing multiple deployments
+
+To juggle between multiple local deployments (e.g: to run different flavors of Open Food Facts on the same host), you will need:
+
+* Multiple `.env` files (one per deployment), such as:
+  * `.env.off` : configuration for Open Food Facts dev env.
+  * `.env.off-pro` : configuration for Open Food Facts Producer's Platform dev env.
+  * `.env.obf`: configuration for Open Beauty Facts dev env.
+  * `.env.opff`: configuration for Open Ped Food Facts dev env.
+* `COMPOSE_PROJECT_NAME` set to different values in each `.env` file, so that container names across deployments are unique.
+* `FRONTEND_PORT` set to different values in each `.env` file, so that frontend containers don't port-conflict with each other.
+
+To switch between configurations, set `ENV_FILE` before running `make` commands:
+
+```
+ENV_FILE=.env.off-pro make up # starts the OFF Producer's Platform containers.
+ENV_FILE=.env.obf     make up # starts the OBF containers.
+ENV_FILE=.env.opff    make up # starts the OPFF containers.
+```
+
+or export it to keep it for a while:
+
+```
+export ENV_FILE=.env.off # going to work on OFF for a while
+make up
+make restart
+make down
+make log
+```
+
+A good strategy is to have multiple terminals open, one for each deployment:
+
+*   `off` \[Terminal 1]:
+
+    ```
+    export ENV_FILE=.env.off
+    make up
+    ```
+*   `off-pro` \[Terminal 2]:
+
+    ```
+    export ENV_FILE=.env.off-pro
+    make up
+    ```
+*   `obf` \[Terminal 3]:
+
+    ```
+    export ENV_FILE=.env.obf
+    make up
+    ```
+*   `opff` \[Terminal 3]:
+
+    ```
+    export ENV_FILE=.env.opff
+    make up
+    ```
+
+**Note:** the above case of 4 deployments is _**a bit ambitious**_, since ProductOpener's `backend` container takes about \~6GB of RAM to run, meaning that the above 4 deployments would require a total of 24GB of RAM available.

how-to-guides/pro-development.md

@@ -0,0 +1,41 @@
+# Developing on the producers platform
+
+Here is how to develop for the producers platform using docker.
+
+It suppose [you already have setup docker for dev](../introduction/dev-environment-quick-start-guide.md).
+
+You should have two kind of shell:
+
+* the shell for openfoodfacts
+* the shell for openfoodfacts-pro, this is a shell where you have source the `setenv-pro.sh`, that is you run `. setenv-pro.sh`. Your prompt, should now contains a `(pro)` to recall you you are in producers environment. (this simply sets some environment variables that will overides the one in .env)
+
+To develop, on producers plateform, you can then us a shell for openfoodfacts-pro and simply do a `make dev` and everything as usual.
+
+If you need to work on product import/export, or interacting with public platform, you have to start postgres and the minion on off side. That is, in a _non pro_ shell, run `docker-compose up postgres minion mongodb`.
+
+Note that the setup does not currently support running the http server for both public and pro platform at the same time. So as you need the public platform:
+
+* in your _pro shell_, run a `docker-compose stop backend`
+* in your _non pro shell_, run a `docker-compose up backend` Now `openfoodfacts.localhost` is the public database. Of course, do this inside-out to access the pro http server.
+
+Note that if you [use direnv](use-direnv.md), it should be fine, if you did not redefine variables set by `setenv-pro.sh`.
+
+An explanation of the setup can be found at [pro-dev-setup.md](../explanations/pro-dev-setup.md)
+
+If you want to see state of tasks, you can run:
+
+```
+docker-compose exec minion /opt/product-opener/scripts/minion.pl  minion job
+```
+
+(add --help to see all options), or refer to https://docs.mojolicious.org/Minion/Command/minion/job
+
+You may also inspect database by running:
+
+```
+docker-compose exec  postgres psql -U productopener -W minion
+```
+
+(password is given by `POSTGRES_PASSWORD` in `.env` and defaults to `productopener`)
+
+Inspecting table minion, should help.

how-to-guides/use-direnv.md

@@ -0,0 +1,28 @@
+# Use direnv
+
+As a developper, it can be better not to think too much about setting right env variables as you enter a project. [`direnv`](https://direnv.net/) aims at providing a solution.
+
+As a quick guide as an openfoodfacts developper:
+
+* install direnv on your system using usual package manager
+*   in your .bashrc add:
+
+    ```bash
+    # direnv
+    eval "$(direnv hook bash)"
+    ```
+
+    you have adapt the direnv line according to what you use, see [direnv doc](https://direnv.net/docs/hook.html)
+* In your project directory add a file, where you superseed variables from `.env` that you wan't to
+
+```
+echo "setting up docker-compose env"
+export DOCKER_BUILDKIT=1
+export USER_UID=${UID}
+export USER_UID=$(id -g)
+```
+
+* in project directory, run `direnv allow .`
+* in a new shell:
+  * go in project directory
+  * you should have direnv trigger and load your variables