replace quickstart setup by deployment examples for helm and compose (#…

…622) * move quickstart to examples/compose * add helm installation option * add opensiem example * add istio installation * add istio gateway and virtual service * use ingress in favor of istio * update chart version * update documentation for utilizing fda and ucl in compose example for little bit more consistency --------- Co-authored-by: djkhl <[email protected]>
fkie-cad · Jul 19, 2024 · 6c77b88 · 6c77b88
1 parent f06e3e4
commit 6c77b88
Show file tree

Hide file tree

Showing 82 changed files with 523 additions and 139 deletions.
diff --git a/.github/secret_scanning.yml b/.github/secret_scanning.yml
@@ -1,4 +1,4 @@
 paths-ignore:
-  - "quickstart/exampledata/config/credentials.yml"
-  - "quickstart/exampledata/config/nginx/mtls.conf.d/*.key"
-  - "quickstart/exampledata/config/nginx/mtls.conf.d/*.crt"
+  - "examples/exampledata/config/credentials.yml"
+  - "examples/exampledata/config/nginx/mtls.conf.d/*.key"
+  - "examples/exampledata/config/nginx/mtls.conf.d/*.crt"
diff --git a/.gitignore b/.gitignore
@@ -25,3 +25,4 @@ experiments
 **/_static/*.xlsx
 logprep.log
 /charts/logprep/charts
+examples/k8s/charts
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,12 +3,15 @@
 ## next release
 ### Breaking
 ### Features
+
 ### Improvements
 
 * a result object was added to processors and pipelines
   * each processor returns an object including the processor name, generated extra_data, warnings
     and errors
   * the pipeline returns an object with the list of all processor result objects
+* add kubernetes opensiem deployment example
+* move quickstart setup to compose example
 
 ### Bugfix
 
@@ -316,7 +319,7 @@
 
 * add possibility to convert hex to int in `calculator` processor with new added function `from_hex`
 * add metrics on rule level
-* add grafana example dashboards under `quickstart/exampledata/config/grafana/dashboards`
+* add grafana example dashboards under `examples/exampledata/config/grafana/dashboards`
 * add new configuration field `id` for all rules to identify rules in metrics and logs
   - if no `id` is given, the `id` will be generated in a stable way
   - add verification of rule `id` uniqueness on processor level over both rule trees to ensure metrics are counted correctly on rule level

diff --git a/README.md b/README.md
@@ -29,7 +29,7 @@ allowing further applications besides log handling.
 This readme provides basic information about the following topics:
 - [About Logprep](#about-logprep)
 - [Getting Started](https://logprep.readthedocs.io/en/latest/getting_started.html)
-- [Docker Quickstart](https://logprep.readthedocs.io/en/latest/getting_started.html#logprep-quickstart-environment)
+- [Deployment Examples](https://logprep.readthedocs.io/en/latest/examples/index.html)
 - [Event Generation](https://logprep.readthedocs.io/en/latest/user_manual/execution.html#event-generation)
 - [Documentation](https://logprep.readthedocs.io/en/latest)
 - [Contributing](#contributing)

diff --git a/charts/logprep/Chart.yaml b/charts/logprep/Chart.yaml
@@ -6,7 +6,7 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: "13.2.0"
+version: "13.2.1"
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to

diff --git a/charts/logprep/templates/deployment.yaml b/charts/logprep/templates/deployment.yaml
@@ -2,7 +2,7 @@
 apiVersion: apps/v1
 kind: Deployment
 metadata:
-  name: {{ include "logprep.name" . }}
+  name: {{  include "logprep.fullname" . }}
   labels:
     {{- include "logprep.labels" . | nindent 4 }}
 spec:
@@ -14,6 +14,7 @@ spec:
       {{- include "logprep.selectorLabels" . | nindent 6 }}
   template:
     metadata:
+      name: {{ include "logprep.fullname" . }}
       labels:
         {{- include "logprep.labels" . | nindent 8 }}
       annotations:

diff --git a/charts/logprep/values.yaml b/charts/logprep/values.yaml
@@ -118,6 +118,9 @@ logger:
 #
 # See https://logprep.readthedocs.io/en/latest/user_manual/configuration/input.html
 # for available configuration options.
+# Note:
+# For the `http_input` endpoints you have to add the endpoint `/health: plaintext` to ensure
+# readiness probes are working.
 input: {}
 
 # The logprep output connector configuration

diff --git a/doc/source/development/programaticly_start_logprep.rst b/doc/source/development/programaticly_start_logprep.rst
@@ -23,10 +23,10 @@ An example with input connector and preprocessors could look like this:
                 "predetector": {
                     "type": "pre_detector",
                     "specific_rules": [
-                        "quickstart/exampledata/rules/pre_detector/specific"
+                        "examples/exampledata/rules/pre_detector/specific"
                     ],
                     "generic_rules": [
-                        "quickstart/exampledata/rules/pre_detector/generic"
+                        "examples/exampledata/rules/pre_detector/generic"
                     ],
                     "pre_detector_topic": "output_topic"
                 }
@@ -61,10 +61,10 @@ An example without input connector and preprocessors could look like this:
                 "predetector": {
                     "type": "pre_detector",
                     "specific_rules": [
-                        "quickstart/exampledata/rules/pre_detector/specific"
+                        "examples/exampledata/rules/pre_detector/specific"
                     ],
                     "generic_rules": [
-                        "quickstart/exampledata/rules/pre_detector/generic"
+                        "examples/exampledata/rules/pre_detector/generic"
                     ],
                     "pre_detector_topic": "output_topic"
                 }

diff --git a/doc/source/examples/compose.rst b/doc/source/examples/compose.rst
@@ -5,7 +5,7 @@ Docker Compose Example Deployment
 To demonstrate the functionality of logprep this repo comes with a complete `kafka`, `logprep` and
 `opensearch` stack.
 To get it running `docker` with compose support must be first installed.
-The docker compose file is located in the directory `quickstart`.
+The docker compose file is located in the directory `examples/compose`.
 A prerequisite is to run `sysctl -w vm.max_map_count=262144`, otherwise Opensearch might not
 properly start.
 
@@ -14,7 +14,7 @@ The environment can either be started with a Logprep container or without one:
 Run without Logprep Container (default)
 ---------------------------------------
 
-  1. Run from within the `quickstart` directory:
+  1. Run from within the `examples/compose` directory:
 
      .. code-block:: bash
 
@@ -25,22 +25,22 @@ Run without Logprep Container (default)
 
      .. code-block:: bash
 
-      logprep run quickstart/exampledata/config/pipeline.yml
+      logprep run examples/exampledata/config/pipeline.yml
 
     If logprep is run with the metrics enabled, the necessary environment variable has to be set first:
 
     .. code-block:: bash
 
       export PROMETHEUS_MULTIPROC_DIR="tmp/logprep"
-      logprep run quickstart/exampledata/config/pipeline.yml
+      logprep run examples/exampledata/config/pipeline.yml
 
 
 
 
 Run with Logprep Container
 --------------------------
 
-  * Run from within the `quickstart` directory:
+  * Run from within the `examples/compose` directory:
 
     .. code-block:: bash
 
@@ -50,7 +50,7 @@ Run with Logprep Container
 Run with getting config from http server with basic authentication
 ------------------------------------------------------------------
 
-  * Run from within the `quickstart` directory:
+  * Run from within the `examples/compose` directory:
 
     .. code-block:: bash
 
@@ -60,14 +60,14 @@ Run with getting config from http server with basic authentication
 
     .. code-block:: bash
 
-      export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
+      export LOGPREP_CREDENTIALS_FILE="examples/exampledata/config/credentials.yml"
       logprep run http://localhost:8081/config/pipeline.yml
 
 
 Run with getting config from http server with mTLS authentication
 -----------------------------------------------------------------
 
-  * Run from within the `quickstart` directory:
+  * Run from within the `examples/compose` directory:
 
     .. code-block:: bash
 
@@ -77,12 +77,12 @@ Run with getting config from http server with mTLS authentication
 
     .. code-block:: bash
 
-      export LOGPREP_CREDENTIALS_FILE="quickstart/exampledata/config/credentials.yml"
+      export LOGPREP_CREDENTIALS_FILE="examples/exampledata/config/credentials.yml"
       logprep run https://localhost:8082/config/pipeline.yml
 
 
-Interacting with the Quickstart Environment
--------------------------------------------
+Interacting with the Compose Environment
+----------------------------------------
 
 The start up takes a few seconds to complete, but once everything is up
 and running it is possible to write JSON events into Kafka and read the processed events in
@@ -111,9 +111,9 @@ UCL Postgres:          `localhost:5432`  ucl                       ucl
 ====================== ================= ========================  =======================
 
 The example rules that are used in the docker instance of Logprep can be found
-in `quickstart/exampledata/rules`.
+in `examples/exampledata/rules`.
 Example events that trigger for the example rules can be found in
-`quickstart/exampledata/input_logdata/logclass/test_input.jsonl`.
+`examples/exampledata/input_logdata/logclass/test_input.jsonl`.
 These events can be added to Kafka with the following command:
 
 .. code-block:: bash
@@ -133,12 +133,17 @@ Utilizing FDA and UCL
 If you want to try out the FDA and UCL you first have to do some preparations.
 
 
-0. Run the quickstart setup with the :code:`oauth2` profile:
-   :code:`docker compose --profile oauth2 up -d`.
+0. Run the example compose setup with the :code:`oauth2` profile:
+
+.. code-block:: bash
+
+  docker compose --profile oauth2 up -d.
+
+
 1. Sign into the keycloak admin panel and create a logprep user in the :code:`logprep` realm.
    Make sure that the user is part of the :code:`logprep-admin` group and has a password. If you
    choose a password other than :code:`logprep` you have to update the credentials file
-   :code:`quickstart/exampledata/config/credentials.yml`, such that the password of
+   :code:`examples/exampledata/config/credentials.yml`, such that the password of
    :code:`http://localhost:3001` and :code:`http://localhost:3002` reflects your choice.
 2. You have to login to the FDA with the previously created user and create a release, as well
    as your first logclass. It is also necessary to add an example event to this logclass in order
@@ -148,19 +153,19 @@ If you want to try out the FDA and UCL you first have to do some preparations.
    your created logprep user and then configure required Use-Cases.
    At the current moment these configuration are not yet processed by logprep though, as the ucl
    only provides a mock endpoint which doesn't contain your Use-Case configurations.
-4. Set the env :code:`LOGPREP_CREDENTIALS_FILE` to :code:`quickstart/exampledata/config/credentials.yml`
+4. Set the env and run logprep
+
+  .. code-block:: bash
+
+    export LOGPREP_CREDENTIALS_FILE="examples/exampledata/config/credentials.yml"
+    logprep run examples/exampledata/config/pipeline.yml "http://localhost:3002/api/v1/pipelines?stage=prod&logclass=ExampleClass" "http://localhost:3001/api/v1/general-predetection"
 
-Once you have set everything up you can run logprep with the following command.
 Just consider that the first :code:`pipeline.yml` argument is used to define a proper :code:`input`
 and :code:`output` as those are not part of the FDA/UCL output. Also, in the second argument
 you should ensure that the :code:`stage` and :code:`loglcass` are set properly.
 
-.. code-block:: bash
-
-     logprep run quickstart/exampledata/config/pipeline.yml "http://localhost:3002/api/v1/pipelines?stage=prod&logclass=ExampleClass" "http://localhost:3001/api/v1/general-predetection"
-
 .. note::
 
-     If you did use the quickstart setup before and run into problems it is advised to first pull
+     If you did use the example compose setup before and run into problems it is advised to first pull
      all images again to update them to the latest version:
-     :code:`docker compose -f ./quickstart/docker-compose.yml pull`.
+     :code:`docker compose -f ./examples/compose/docker-compose.yml pull`.
diff --git a/doc/source/examples/minikube.rst b/doc/source/examples/minikube.rst
@@ -45,15 +45,26 @@ with the following commands:
     sudo mv linux-amd64/helm /usr/local/bin/helm
     sudo chmod +x /usr/local/bin/helm
 
+.. code-block:: bash
+    :caption: add helm repositories
+
+    helm repo add bitnami https://charts.bitnami.com/bitnami
+
 .. code-block:: bash
     :caption: Configure and start minikube
     
     minikube config set driver docker
+    minikube config set cpus 16 
+    minikube config set memory 16GB
+    minikube addons enable ingress
     minikube start
 
 Deploy the example
 ------------------
 
+The following steps install the actual opensiem example on the minikube cluster.
+It will install 
+
 At first you have to install the prometheus PodMonitor CRD:
 
 .. code-block:: bash
@@ -62,9 +73,56 @@ At first you have to install the prometheus PodMonitor CRD:
     kubectl apply -f https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/charts/crds/crds/crd-podmonitors.yaml
 
 
-Next you can install logprep using:
+Then you have to update and build the helm subcharts repository:
 
 .. code-block:: bash
-    :caption: Install logprep
+    :caption: Add the bitnami helm repository
+
+    helm dependencies update ./examples/k8s
+    helm dependencies build ./examples/k8s
+
+Next you are ready to install the opensiem example using:
 
-    helm install logprep charts/logprep
+.. code-block:: bash
+    :caption: Install opensiem
+
+    helm install opensiem examples/k8s
+
+Make the cluster locally resolvable:
+
+.. code-block:: bash
+    :caption: add hosts entry to resolve the cluster
+
+    echo "$( minikube ip ) connector.opensiem dashboards.opensiem grafana.opensiem" | sudo tee -a /etc/hosts
+
+Test the defined ingresses:
+
+.. code-block:: bash
+    :caption: Test the opensiem example ingress
+
+    curl -v http://connector.opensiem/health
+    curl -v http://dashboards.opensiem
+
+Test the opensiem connector:
+
+.. code-block:: bash
+    :caption: Test the opensiem example connector
+
+    ❯ logprep generate http --input-dir ./examples/exampledata/input_logdata/ --target-url http://connector.opensiem --events 100 --batch-size 10
+    
+    2024-07-17 11:15:35 301643 Generator  INFO    : Log level set to 'NOTSET'
+    2024-07-17 11:15:35 301643 Generator  INFO    : Started Data Processing
+    2024-07-17 11:15:35 301643 Input      INFO    : Reading input dataset and creating temporary event collections in: '/tmp/logprep_a51e1vh6'
+    2024-07-17 11:15:35 301643 Input      INFO    : Preparing data took: 0.0042 seconds
+    2024-07-17 11:15:35 301643 Input      INFO    : Cleaned up temp dir: '/tmp/logprep_a51e1vh6'
+    2024-07-17 11:15:35 301643 Generator  INFO    : Completed with following statistics: {
+        "Number of failed events": 0,
+        "Number of successfull events": 100,
+        "Requests Connection Errors": 0,
+        "Requests Timeouts": 0,
+        "Requests http status 200": 10,
+        "Requests total": 10
+    }
+    2024-07-17 11:15:35 301643 Generator  INFO    : Execution time: 0.067013 seconds
+
+open your browser and go to `http://dashboards.opensiem`_ to see the generated data in the opensearch dashboards.
diff --git a/doc/source/installation.rst b/doc/source/installation.rst
@@ -61,4 +61,26 @@ Helm
 
 This option can be used to deploy logprep on a kubernetes cluster.
 
-tbd
+At first you have to install the prometheus PodMonitor CRD:
+
+.. code-block:: bash
+    :caption: Install the prometheus PodMonitor CRD
+
+    kubectl apply -f https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/charts/crds/crds/crd-podmonitors.yaml
+
+
+To install latest stable release:
+
+..  code-block:: bash
+
+   helm repo add logprep https://fkie-cad.github.io/Logprep
+   helm install logprep logprep/logprep
+
+
+To install from cloned github repository:
+
+..  code-block:: bash
+
+   git clone https://github.com/fkie-cad/Logprep.git
+   cd Logprep
+   helm install logprep charts/logprep