HELK Fortigate Firewall Syslogs Field Extractions, how to improve?

[beachcondo]

Describe the problem

Hello fellow HELK users, I could use some syslog/firewall help with HELK.

I am running the HELK, with winlogbeats with the Sysmon dumping to Kafka flawlessly.

My problem is with my Forti firewall, and it having to use Syslog.

I am able to receive the syslogs, and ingest them to an index. The data is ugly as can be to the logs-indexme*, which is another thing I wouldn't want it to be.

I have tried finding other solutions on the internet, but every time I find them, I just break it all and have to end up reverting after several lost hours. This has occurred multiple times.

The fixes I have tried has been changing input.conf and output.conf and also added in filters to filter files, which I am lost if I am putting them in the correct place, it just breaks everything, and decided to better leave it the way it is for now with it at least ingesting.

Get operating system and version
for linux (except Mac) use:  
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal


Get disk space, memory, processor cores, and docker storage  
Docker Space:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       196G   39G  147G  21% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:             15          11           0           0           4           3
Swap:             3           0           3

Cores:
4

Get output of the HELK docker containers:  
CONTAINER ID   IMAGE                                                 COMMAND                  CREATED      STATUS             PORTS
                                                                                                                                                                                                                                                                                       NAMES
2092979b3cd9   confluentinc/ksqldb-cli:latest                        "/bin/sh"                8 days ago   Up About an hour                                                                                                                                                                                                                                                                                                                                                                   helk-ksql-cli
776b063d4784   confluentinc/ksqldb-server:latest                     "/usr/bin/docker/run"    8 days ago   Up About an hour   0.0.0.0:8088->8088/tcp, :::8088->8088/tcp                                                                                                                                                                                                                                                                                                                       helk-ksql-server
75f273f78de0   otrf/helk-kafka-broker:2.4.0                          "./kafka-entrypoint.…"   8 days ago   Up About an hour   0.0.0.0:9092->9092/tcp, :::9092->9092/tcp                                                                                                                                                                                                                                                                                                                       helk-kafka-broker
7b315bb40dbb   otrf/helk-spark-worker:2.4.5                          "./spark-worker-entr…"   8 days ago   Up About an hour                                                                                                                                                                                                                                                                                                                                                                   helk-spark-worker
076850dc6b55   otrf/helk-zookeeper:2.4.0                             "./zookeeper-entrypo…"   8 days ago   Up About an hour   2181/tcp, 2888/tcp, 3888/tcp                                                                                                                                                                                                                                                                                                                                    helk-zookeeper
1aff5c7bdb3e   otrf/helk-spark-master:2.4.5                          "./spark-master-entr…"   8 days ago   Up About an hour   7077/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp
                                                                                                                                                                                                                                                                                       helk-spark-master
61028c083109   docker_helk-jupyter                                   "/opt/jupyter/script…"   8 days ago   Up About an hour   8000/tcp, 8888/tcp                                                                                                                                                                                                                                                                                                                                              helk-jupyter
e2b2d0f94e9b   otrf/helk-elastalert:latest                           "./elastalert-entryp…"   8 days ago   Up About an hour                                                                                                                                                                                                                                                                                                                                                                   helk-elastalert
6959eed9c5ef   otrf/helk-nginx:0.3.0                                 "/opt/helk/scripts/n…"   8 days ago   Up About an hour   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp                                                                                                                                                                                                                                                                                        helk-nginx
469c5ebf128d   otrf/helk-logstash:7.6.2.1                            "/usr/share/logstash…"   8 days ago   Up About an hour   0.0.0.0:3515->3515/tcp, :::3515->3515/tcp, 0.0.0.0:5044->5044/tcp, :::5044->5044/tcp, 0.0.0.0:5514->5514/tcp, 0.0.0.0:5514->5514/udp, :::5514->5514/tcp, :::5514->5514/udp, 0.0.0.0:8515-8516->8515-8516/tcp, :::8515-8516->8515-8516/tcp, 0.0.0.0:8531->8531/tcp, :::8531->8531/tcp, 0.0.0.0:8515-8516->8515-8516/udp, :::8515-8516->8515-8516/udp, 9600/tcp   helk-logstash
00d82738009d   docker.elastic.co/kibana/kibana:7.6.2                 "/usr/share/kibana/s…"   8 days ago   Up About an hour   5601/tcp                                                                                                                                                                                                                                                                                                                                                        helk-kibana
44a40e962088   docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/share/elastics…"   8 days ago   Up About an hour   9200/tcp, 9300/tcp                                                                                                                                                                                                                                                                                                                                              helk-elasticsearch

What version of HELK are you using

run the command from within the HELK repo run git log -1 --oneline

ad752b2 (HEAD -> master, origin/master, origin/HEAD) Update jvm.options (#563)

Any additional context or input you have

pictures, comments, etc.