Merge pull request #280 from linea-it/integrate_orchestration

Integrate orchestration

.devcontainer/devcontainer.json

@@ -1,7 +1,7 @@
 // For format details, see https://aka.ms/devcontainer.json. For config options, see the
 // README at: https://github.com/devcontainers/templates/tree/main/src/postgres
 {
-	"name": "Python 3 & PostgreSQL",
+	"name": "Pz Server",
 	"dockerComposeFile": [
 		"../docker-compose.yml",
 		"docker-compose.yml"

.devcontainer/docker-compose.yml

@@ -5,10 +5,15 @@ services:
     build:
       context: .
       dockerfile: .devcontainer/Dockerfile
+      # dockerfile: backend/Dockerfile
     volumes:
       - ..:/workspaces:cached
-      - ./archive/log/backend:/archive/log
-      - ./archive/data:/archive/data
+      - ./backend:/app:cached
+      - ./archive/log/backend:/archive/log:cached
+      - ./archive/data:/archive/data:cached
+      - ./orchestration/pipelines:/pipelines:cached
+      - ./orchestration/processes:/processes:cached
+      - ./orchestration/datasets:/datasets:cached
 
     # Overrides default command so things don't shut down after the process ends.
     command: sleep infinity

.gitignore

@@ -12,11 +12,24 @@ venv.bak/
 pg_data/
 pg_backups/
 
-# Ngnix 
+# Ngnix
 nginx.conf
 
-# Docker Compose 
+# Docker Compose
 docker-compose.yml
 .env.local
 
-saml2
+# Orchestration
+orchestration/db
+orchestration/processes
+orchestration/logs
+orchestration/rabbitmq/*
+!orchestration/rabbitmq/enabled_plugins
+!orchestration/docker-compose.yml
+!orchestration/nginx.conf
+
+saml2
+
+*.pyc
+__pycache__
+_backup

.gitmodules

@@ -0,0 +1,4 @@
+[submodule "orchestration/pipelines"]
+	path = orchestration/pipelines
+    url = https://github.com/linea-it/pzserver_pipelines
+    branch = training_set_maker

README.md

@@ -12,7 +12,6 @@ Clone the repository and access the directory:
 ```bash
 git clone https://github.com/linea-it/pzserver_app.git  
 cd pzserver_app
-mkdir -p archive/data archive/log/backend
 ```
 
 Copy the file `docker-compose-development.yml` and rename to `docker-compose.yml`
@@ -40,32 +39,32 @@ and update it in the `docker-compose.yml` file if necessary (if it is not the us
 Now start the database service. It is important that the first time the database service is turned on alone, in this step postgresql will create the database and the user based on the settings `POSTGRES_USER`, `POSTGRES_PASSWORD` and `POSTGRES_DB`.
 
 ```bash
-docker-compose up database
+docker compose up database
 ```
 
-Wait for the message `database system is ready to accept connections` and then close the service with the `CTRL + C` keys or `docker-compose stop database` in another terminal.
+Wait for the message `database system is ready to accept connections` and then close the service with the `CTRL + C` keys or `docker compose stop database` in another terminal.
 
 Now start the backend service. As this is the first time, the base image will be pulled and the container will be built, this may take a while.
 
 ```bash
-docker-compose up backend
+docker compose up backend
 ```
 If everything goes normally the last message will be something like `... spawned uWSGI worker x (pid: xx, cores: x)`
 
 Shut down the backend service to change one of the Django variables.
 
-To terminate use `CTRL + C` or `docker-compose stop`.
+To terminate use `CTRL + C` or `docker compose stop`.
 
 With the services turned off, let's run a command in the backend container to generate a SECRET for Django.
 
 ```bash
-docker-compose run backend python -c "import secrets; print(secrets.token_urlsafe())"
+docker compose run backend python -c "import secrets; print(secrets.token_urlsafe())"
 ```
 
 This is the output of the command:
 
 ```bash
-$ docker-compose run backend python -c "import secrets; print(secrets.token_urlsafe())"
+$ docker compose run backend python -c "import secrets; print(secrets.token_urlsafe())"
 Creating pzserver_backend_run ... done
 6klbHhaeA6J2imKt9AVVgS5yl9mCWoiQqrfUV469DLA
 ```
@@ -75,21 +74,21 @@ Copy the generated key and replace the `SECRET` variable value in the `.env` fil
 Create the Django superuser.
 
 ```bash
-docker-compose run backend python manage.py createsuperuser
+docker compose run backend python manage.py createsuperuser
 ```
 
 Import the application's initial data using the following command:
 
 ```bash
-docker-compose run backend python manage.py loaddata initial_data
+docker compose run backend python manage.py loaddata initial_data
 ```
 
 This `loaddata` command will insert some basic records into the database for the application to work. these records are in the `core/fixtures/initial_data.yaml` file.
 
 Now install the Frontend dependencies by running the `yarn` command. As this is the first time starting this container, the base image will be pulled, which may take a while.
 
 ```bash
-docker-compose run frontend yarn
+docker compose run frontend yarn
 ```
 
 This command will create the directory `pzserver/frontend/node_modules` if you have any problem with dependencies remove this directory and run the command again.
@@ -98,12 +97,10 @@ In the development environment it is not necessary to change Ngnix settings.
 But if a local change is needed, copy the `nginx_development.conf` file to `nginx.conf`
 Also change the `docker-compose.yml` file in the ngnix service at the line `- ./nginx_development.conf:/etc/nginx/conf.d/default.conf:ro`. In this way, the ngnix.conf file represents your local environment, if you make any modifications that are necessary for the project, copy this modification to the template file, as the nginx.conf file is not part of the repository.
 
-Once this is done, the development environment setup process is complete.
-
-Finally, to start the whole application: 
+Finally, to start the whole application:
 
 ``` bash
-docker-compose up 
+docker compose up
 ```
 
 ### Setting Up a New Application to manage authentication
@@ -116,22 +113,79 @@ Go to Django ADMIN (for local installation, open a web browser and go to the URL
 - `client_type` should be set to confidential
 - `authorization_grant_type` should be set to **'Resource owner password-based'**
 - `name` can be set to whatever you'd like
-- checkbox for `Skip authorization` should remain unchecked 
-- `Algorithm`: keep the default option (No OIDC support) 
+- checkbox for `Skip authorization` should remain unchecked
+- `Algorithm`: keep the default option (No OIDC support)
 
 > **WARNING**: As mentioned above, **BEFORE** clicking on the SAVE button, you must edit the configuration files (**.env** and **.env.local**) and change the variables `DJANGO_OAUTH_CLIENT_ID` and `DJANGO_OAUTH_CLIENT_SECRET` in both files according to the values of `client_id` and `client_secret` respectively. only after editing the configuration files, the `SAVE` button must be pressed.
 
 ![Adding new application](images/new_app.png)
 
-The installation is done, you can now test the newly configured application.
+The installation is done. To test the newly configured application, restart the service by pressing `Ctrl+C` in the terminal, then execute:  
+
+```
+docker compose up -d 
+```
+
+And then, go to `https://localhost` to open the application.   
+
+
+
+### Orchestration setup
+
+The Pz Server uses [orchestration](https://github.com/linea-it/orchestration/) to process its pipelines and for this you need to configure it:
+
+``` bash
+mkdir orchestration/db orchestration/logs orchestration/processes
+```
+
+``` bash
+cp docker-compose-development-orch.yml docker-compose.yml
+docker network create orchestration-network  # create internal network
+```
+
+Loading pipelines (submodules):
+
+``` bash
+git submodule init
+git submodule update
+```
+
+Enter the orchestration-api container:
+``` bash
+docker compose run orchestration-api bash
+```
+
+Inside the container, create the database and an admin user:
+``` bash
+python manage.py migrate
+python manage.py createsuperuser
+```
+
+Still inside the container, installing pipelines (confirm default directories with 'yes'):
+``` bash
+cd /pipelines
+./setup.sh
+```
+
+Exit the container and start orchestration services:
+``` bash
+docker compose up orchestrator
+```
+
+And then follow the steps to create an authentication application ([step by step](https://github.com/linea-it/orchestration/?tab=readme-ov-file#how-to-use-using-client-credential)) just by changing the url from  `http://localhost/admin/oauth2_provider/application/add/` to `http://localhost:8088/admin/oauth2_provider/application/add/` and using the admin user created previously. Note when creating an authentication application, we must change the `ORCHEST_CLIENT_ID` and `ORCHEST_CLIENT_SECRET` in the `.env` with the client_id and secret_id values ​​respectively.
+
+All that remains is to modify the ORCHEST_URL variable in the `.env` with the value http://orchestrator
+``` bash
+ORCHEST_URL=http://orchestrator
+```
 
 ### Some example commands
 
 Turn on background environment (if you have the application already running on the terminal, stop it with `CTRL + C` keys and up ir again, but in the background using `-d` argument):
 
 
 ```bash
-docker-compose up -d
+docker compose up -d
 ```
 
 Access in the browser:
@@ -142,41 +196,41 @@ Access in the browser:
 Turn off all environment:
 
 ```bash
-docker-compose stop
+docker compose stop
 ```
 
 Restart all environment:
 
 ```bash
-docker-compose stop && docker-compose up -d
+docker compose stop && docker compose up -d
 ```
 
 Run a terminate on one of the services
 
 ```bash
 # with the service turned on
-docker-compose exec backend bash
+docker compose exec backend bash
 # with the service turned off
-docker-compose run backend bash
+docker compose run backend bash
 ```
 
 Access database with psql:
 
 ```bash
 # Use the credentials that are in the .env
-docker-compose exec database psql -h localhost -U <username> -d <database>
+docker compose exec database psql -h localhost -U <username> -d <database>
 ```
 
 Add libraries to frontend using yarn:
 
 ``` bash
-docker-compose run frontend yarn add <library>
+docker compose run frontend yarn add <library>
 ```
 
 Check front-end changes before pushing new commits to the remote repository (it is recommended to build the frontend to prevent errors with ESlint from disrupting the Pull Request process):
 
 ``` bash
-docker-compose run frontend yarn build
+docker compose run frontend yarn build
 ```
 
 ### Manual build of images and push to docker hub
@@ -206,31 +260,31 @@ docker push linea/pzserver:frontend_<commit_hash>
 run all tests
 
 ```bash
-docker-compose exec backend pytest
+docker compose exec backend pytest
 ```
 
 run all tests with coverage, Check local coverage in localhost/coverage
 
 ```bash
-docker-compose exec backend pytest --cov=./ --cov-report=html
+docker compose exec backend pytest --cov=./ --cov-report=html
 ```
 
 run only a file
 
 ```bash
-docker-compose exec backend pytest core/test/test_product_file.py
+docker compose exec backend pytest core/test/test_product_file.py
 ```
 
 run only a class
 
 ```bash
-docker-compose exec backend pytest core/test/test_product_file.py::ProductFileListCreateAPIViewTestCase
+docker compose exec backend pytest core/test/test_product_file.py::ProductFileListCreateAPIViewTestCase
 ```
 
 run single test method
 
 ```bash
-docker-compose exec backend pytest core/test/test_product_file.py::ProductFileListCreateAPIViewTestCase::test_list_product_file
+docker compose exec backend pytest core/test/test_product_file.py::ProductFileListCreateAPIViewTestCase::test_list_product_file
 ```
 
 ## Enable authentication via LIneA Satosa (Github)
@@ -322,16 +376,16 @@ Usually the changes are in ngnix volumes and port.
 
 Create an `.env` file based on the `env_template` file and edit the database access variables.
 
-Wait for the message `database system is ready to accept connections` and then close the service with the `CTRL + C` keys or `docker-compose stop database` in another terminal.
+Wait for the message `database system is ready to accept connections` and then close the service with the `CTRL + C` keys or `docker compose stop database` in another terminal.
 
 ```bash
-docker-compose up database
+docker compose up database
 ```
 
 Start the backend service and wait for the `Booting worker with pid...` message.
 
 ```bash
-docker-compose up backend
+docker compose up backend
 ```
 
 Shutdown the backend service and change the Django variables.
@@ -342,19 +396,19 @@ In production it is **MANDATORY** to turn off Debug `DEBUG=0`. and change the `S
 With the service turned off, run the command below to generate a SECRET, copy and paste it into the `.env`
 
 ```bash
-docker-compose run backend python -c "import secrets; print(secrets.token_urlsafe())"
+docker compose run backend python -c "import secrets; print(secrets.token_urlsafe())"
 ```
 
 ```bash
-docker-compose run backend python manage.py createsuperuser
+docker compose run backend python manage.py createsuperuser
 ```
 
 Create the Ngnix configuration file `nginx.conf` based on the `nginx_production.conf` file
 
 Start all services
 
 ```bash
-docker-compose up -d
+docker compose up -d
 ```
 
 Configure a URL that points to the machine where it is installed on the port configured for Ngnix in docker-compose.
@@ -376,6 +430,6 @@ Procedure to update the production environment or any other that uses built imag
 
 - Edit the `docker-compose.yml` file and change the frontend and backend images tag.
 - Edit the `.env` file to add new variables or change them if necessary.
-- Pull the new images with the `docker-compose pull` command.
-- Restart services `docker-compose stop && docker-compose up -d`.
+- Pull the new images with the `docker compose pull` command.
+- Restart services `docker compose stop && docker compose up -d`.
 

backend/Dockerfile

@@ -15,6 +15,7 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
     build-essential \
     libpcre3 \
     libpcre3-dev \
+    iputils-ping \
     xmlsec1 \
     libssl-dev \
     libsasl2-dev \