Merge pull request #3 from NethServer/initialWork

First work to ns8-crowdsec

README.md

@@ -1,24 +1,19 @@
 # ns8-crowdsec
 
-This is a template module for [NethServer 8](https://github.com/NethServer/ns8-core).
-To start a new module from it:
+CrowdSec is an open-source and lightweight software that allows you to detect peers with malevolent behaviors and block them from accessing your systems at various levels (infrastructural, system, applicative).
+To achieve this, CrowdSec reads logs from different sources (files, streams ...) to parse, normalize and enrich them before matching them to threats patterns called scenarios.
 
-1. Click on [Use this template](https://github.com/NethServer/ns8-crowdsec/generate).
-   Name your repo with `ns8-` prefix (e.g. `ns8-mymodule`). 
-   Do not end your module name with a number, like ~~`ns8-baaad2`~~!
+ns8-crowdsec installs the LAPI (local api) on the node, composed of agent (to read logs) and a list of collections to parse them (scenarios, parsers...), the notifications can be sent by email.
+When crowdsec detects an IP which is abusing it triggers an alert and then a decision (ban) to block the IP with a bouncer (iptable with IPSET). The bouncer is installed on the node however it could be installed to another Linux machine.
 
-1. An automated initialization workflow starts: wait for its completion.
-   You can follow the run inside the "Actions" tab, the workflow is named "Initial commit"
+## Documentation
 
-1. You can now clone the repository
-
-1. Edit this `README.md` file, by replacing this section with your module
-   description
-
-1. Commit and push your local changes
+https://docs.crowdsec.net/docs/intro
 
 ## Install
 
+**You can install only one crowdsec instance on the node**, other crowdsec instance will conflict with crowdsec1 and fail to start
+
 Instantiate the module with:
 
     add-module ghcr.io/nethserver/crowdsec:latest 1
@@ -28,27 +23,44 @@ Output example:
 
     {"module_id": "crowdsec1", "image_name": "crowdsec", "image_url": "ghcr.io/nethserver/crowdsec:latest"}
 
-## Configure
+## Configure email notification
+
+ns8-crowdsec detects changes in the event smarthost-changed and adapt to the new settings to send email when the alerts or decisions come.
 
-Let's assume that the crowdsec instance is named `crowdsec1`.
+First of all, you need to set the smarthost credentials in the `Settings > Smarthost` panel of the cluster-admin UI
+
+then trigger the action
+
+- `receiver_emails`: all emails account you want to notice when decisions or alert come
+- `helo_host`: set a fully qualified domain name to use the relevant helo with postfix. 
+
+```
+api-cli run configure-email-notification --agent module/crowdsec1 --data - <<EOF
+{
+    "receiver_emails":"[email protected],[email protected]"
+    "helo_host":"myfqdn.domain.com"
+}
+EOF
+```
+
+## Configure
 
-Launch `configure-module`, by setting the following parameters:
-- `<MODULE_PARAM1_NAME>`: <MODULE_PARAM1_DESCRIPTION>
-- `<MODULE_PARAM2_NAME>`: <MODULE_PARAM2_DESCRIPTION>
-- ...
+Let's assume that the crowdsec instance is named `crowdsec1`. Once installed the container is up, nothing is needed, customisation can be done :
 
-Example:
+- by editing a file inside /var/lib/nethserver/crowdsec1/state/crowdsec_config where all config files are stored
+- by editing a file inside the container use the wrapper with the name of the container : `podman exec -ti crowdsec1 bash`
 
-    api-cli run module/crowdsec1/configure-module --data '{}'
+then restart the container : `systemctl restart crowdsec1`
 
-The above command will:
-- start and configure the crowdsec instance
-- (describe configuration process)
-- ...
+### cscli
 
-Send a test HTTP request to the crowdsec backend service:
+crowdsec come with a cli, do `cscli --help`, if you want to know on a specific command  `cscli <command> --help`
 
-    curl http://127.0.0.1/crowdsec/
+- get a glance : `cscli metrics`
+- see the state of installed bouncers : `cscli bouncers list`
+- see the active decisions(ban): `cscli decisions list`
+- see the alerts (discovered IP): `cscli alerts list`
+- see installed collections : `cscli collections list` or `cscli collections list --all`
 
 ## Uninstall
 

build-images.sh

@@ -25,17 +25,16 @@ if ! buildah containers --format "{{.ContainerName}}" | grep -q nodebuilder-crow
 fi
 
 echo "Build static UI files with node..."
-buildah run nodebuilder-crowdsec sh -c "cd /usr/src/ui && yarn install && yarn build"
+buildah run --env="NODE_OPTIONS=--openssl-legacy-provider" nodebuilder-crowdsec sh -c "cd /usr/src/ui && yarn install && yarn build"
 
 # Add imageroot directory to the container image
 buildah add "${container}" imageroot /imageroot
 buildah add "${container}" ui/dist /ui
 # Setup the entrypoint, ask to reserve one TCP port with the label and set a rootless container
 buildah config --entrypoint=/ \
-    --label="org.nethserver.authorizations=traefik@any:routeadm" \
-    --label="org.nethserver.tcp-ports-demand=1" \
-    --label="org.nethserver.rootfull=0" \
-    --label="org.nethserver.images=docker.io/jmalloc/echo-server:latest" \
+    --label="org.nethserver.authorizations=traefik@node:routeadm" \
+    --label="org.nethserver.rootfull=1" \
+    --label="org.nethserver.images=docker.io/crowdsecurity/crowdsec:v1.4.2-debian" \
     "${container}"
 # Commit the image
 buildah commit "${container}" "${repobase}/${reponame}"

imageroot/actions/configure-email-notification/10configure-receivers

@@ -0,0 +1,18 @@
+#!/usr/bin/env python3
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+import json
+import agent
+import sys
+
+# Try to parse the stdin as JSON.
+# If parsing fails, output everything to stderr
+data = json.load(sys.stdin)
+receiver_emails = data["receiver_emails"]
+helo_host = data["helo_host"]
+agent.set_env("RECEIVER_EMAILS", receiver_emails)
+agent.set_env("HELO_HOST", helo_host)

imageroot/actions/configure-email-notification/90restart-crowdsec

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+set -e
+
+exec 1>&2 # Send any output to stderr, to not alter the action response protocol
+
+# we need to start again to expand configuration
+systemctl restart "${MODULE_ID}.service"

imageroot/actions/create-module/01create-secrets

@@ -0,0 +1,15 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+exec 1>&2 # never generate action output. Write to stderr instead.
+set -e
+
+if [[ ! -f secrets/bouncer_keys_firewall.secret ]]; then
+    /usr/bin/mkdir -vp secrets
+    /usr/bin/openssl rand -hex 20 > secrets/bouncer_keys_firewall.secret
+    chmod 400 secrets/bouncer_keys_firewall.secret
+fi

imageroot/actions/create-module/02create-first-configuration

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+set -e
+
+# build folder to store configuration
+mkdir -vp crowdsec_config
+# first build of the configuration
+expand-configuration

imageroot/actions/create-module/03set-log-path

@@ -0,0 +1,17 @@
+#!/usr/bin/env python3
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+import agent
+import os
+
+
+if os.listdir('/run/log/journal'):
+    agent.set_env("CROWDSEC_JOURNAL", "/run/log/journal")
+else:
+    agent.set_env("CROWDSEC_JOURNAL", "/var/log/journal")
+
+agent.dump_env()

imageroot/actions/create-module/05install-bouncers

@@ -0,0 +1,26 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+set -e
+source /etc/os-release
+
+if [[ -x "$(which crowdsec-firewall-bouncer)" ]]; then
+    exit 0 # Skip bouncer installation if executable is already found in PATH
+fi
+
+echo "Install the bouncer:"
+if [[ "${PLATFORM_ID}" == "platform:el9" ]] ; then
+    curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.rpm.sh | bash
+    dnf install -y crowdsec-firewall-bouncer-iptables
+elif [[ "${ID}" == "debian" ]]; then
+    curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | bash
+    apt-get update
+    apt-get -y install crowdsec-firewall-bouncer-iptables
+else
+    echo "The platform is not compatible, we exit"
+    exit 0
+fi

imageroot/actions/create-module/10initialize

@@ -0,0 +1,35 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+set -e
+exec 1>&2 # Send any output to stderr, to not alter the action response protocol
+
+install -m 644 "${AGENT_INSTALL_DIR}/crowdsec.service" "/etc/systemd/system/${MODULE_ID}.service"
+systemctl daemon-reload
+systemctl enable --now "${MODULE_ID}.service"
+
+# Install default collections
+podman exec -ti ${MODULE_ID} cscli hub update
+podman exec -ti ${MODULE_ID} cscli hub upgrade
+
+podman exec -ti ${MODULE_ID} cscli collections install \
+    crowdsecurity/apache2 \
+    crowdsecurity/base-http-scenarios \
+    crowdsecurity/dovecot \
+    crowdsecurity/http-cve \
+    crowdsecurity/linux \
+    crowdsecurity/mariadb \
+    crowdsecurity/nextcloud \
+    crowdsecurity/nginx \
+    crowdsecurity/nginx-proxy-manager \
+    crowdsecurity/pgsql \
+    crowdsecurity/postfix \
+    crowdsecurity/proftpd \
+    crowdsecurity/sshd \
+    crowdsecurity/traefik \
+    crowdsecurity/vsftpd \
+    crowdsecurity/wordpress

imageroot/actions/create-module/30firewall-configuration

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+exec 1>&2 # never generate action output. Write to stderr instead.
+set -e
+
+echo "Create IPSET (ipv4/ipv6) in Firewalld"
+firewall-cmd --permanent --new-ipset=crowdsec6-blacklists --type=hash:net --option="timeout=0" --option="maxelem=150000"
+firewall-cmd --permanent --new-ipset=crowdsec-blacklists --type=hash:net --option="timeout=0" --option="maxelem=150000"
+firewall-cmd --permanent --zone=drop --add-source=ipset:crowdsec-blacklists
+firewall-cmd --permanent --zone=drop --add-source=ipset:crowdsec6-blacklists
+
+echo "Restart firewalld"
+systemctl reload firewalld
+

imageroot/actions/create-module/35register-local-bouncer

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+exec 1>&2 # never generate action output. Write to stderr instead.
+set -e
+
+echo "Register the localhost bouncer"
+secret=$(cat secrets/bouncer_keys_firewall.secret)
+podman exec -ti ${MODULE_ID} cscli bouncers add localhost -k "$secret"

imageroot/actions/create-module/40bouncer-configuration

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+import os
+import json
+import agent
+import agent.tasks
+
+from jinja2 import Environment, FileSystemLoader, select_autoescape
+
+files =["/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml"]
+for f in files:
+    try:
+        os.remove(f)
+    except FileNotFoundError:
+        pass
+
+jenv = Environment(
+    loader=FileSystemLoader(os.getenv("AGENT_INSTALL_DIR")+"/templates"),
+    autoescape=select_autoescape(),
+    keep_trailing_newline=True,
+)
+
+# read secret
+f = open(os.environ["AGENT_STATE_DIR"]+"/secrets/bouncer_keys_firewall.secret")
+secret = f.read()
+f.close()
+
+properties = {
+    "secret": secret
+}
+
+template = jenv.get_template('crowdsec-firewall-bouncer.yaml')
+output = template.render(properties)
+with open("/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml","w") as f:
+    f.write(output)

imageroot/actions/create-module/50start-bouncer

@@ -0,0 +1,12 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+set -e
+exec 1>&2 # Send any output to stderr, to not alter the action response protocol
+
+systemctl enable --now crowdsec-firewall-bouncer.service
+systemctl restart "${MODULE_ID}.service"

imageroot/actions/create-module/70crowdsec-wrapper

@@ -0,0 +1,19 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+exec 1>&2 # never generate action output. Write to stderr instead.
+set -e  # stop at the first error
+
+# Create a wrapper to use the container : cscli metrics
+
+tmpfile=$(mktemp)
+trap "rm -f \${tmpfile}" EXIT
+cat <<EOF >${tmpfile}
+#!/bin/bash
+
+podman exec -ti ${MODULE_ID} cscli "\${@}"
+EOF
+install -v -m 0755 "${tmpfile}" /usr/local/sbin/cscli

imageroot/actions/destroy-module/50destroy

@@ -0,0 +1,16 @@
+#!/bin/bash
+
+#
+# Copyright (C) 2022 Nethesis S.r.l.
+# SPDX-License-Identifier: GPL-3.0-or-later
+#
+
+exec 1>&2 # Send any output to stderr, to not alter the action response protocol
+set -e
+
+#
+# We must stop and clean up what we installed in the "create-module" action
+#
+systemctl disable --now "${MODULE_ID}.service"
+rm -vf "/etc/systemd/system/${MODULE_ID}.service"
+systemctl daemon-reload