Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

all dashboards are not populated #569

Open
splunk-user1 opened this issue Sep 13, 2021 · 0 comments
Open

all dashboards are not populated #569

splunk-user1 opened this issue Sep 13, 2021 · 0 comments

Comments

@splunk-user1
Copy link

Describe the problem

Installed HELK using option 1.
Tried using Mordor dataset.
Populates only mitre dashboards. But NOT global, sysmon or process dashboards
Appreciate clues if I'm missing something

Provide the output of the following commands

Get operating system and version
for linux (except Mac) use:
cat /etc/os-release

NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"

Get disk space, memory, processor cores, and docker storage

Docker Space:
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G   17G   30G  36% /

Memory:
              total        used        free      shared  buff/cache   available
Mem:              7           7           0           0           0           0
Swap:             1           1           0

Cores:
4

Get output of the HELK docker containers:
docker ps --filter "name=helk"

CONTAINER ID   IMAGE                                                 COMMAND                  CREATED      STATUS       PORTS                       
2e92c8ced16d   confluentinc/ksqldb-server:latest                     "/usr/bin/docker/run"    2 days ago   Up 2 hours   0.0.0.0:8088->8088/tcp, :::8
51e0405f3d47   otrf/helk-kafka-broker:2.4.0                          "./kafka-entrypoint.…"   2 days ago   Up 2 hours   0.0.0.0:9092->9092/tcp, :::9
6e21e926114f   otrf/helk-zookeeper:2.4.0                             "./zookeeper-entrypo…"   2 days ago   Up 2 hours   2181/tcp, 2888/tcp, 3888/tcp
2328e19ce870   otrf/helk-logstash:7.6.2.1                            "/usr/share/logstash…"   2 days ago   Up 2 hours   0.0.0.0:3515->3515/tcp, :::3
cf6dadb00ac3   otrf/helk-nginx:0.3.0                                 "/opt/helk/scripts/n…"   2 days ago   Up 2 hours   0.0.0.0:80->80/tcp, :::80->8
52f9430eac38   docker.elastic.co/kibana/kibana:7.6.2                 "/usr/share/kibana/s…"   2 days ago   Up 2 hours   5601/tcp                    
f8a0194c1256   docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/share/elastics…"   2 days ago   Up 2 hours   9200/tcp, 9300/tcp

Provide the HELK installation logs located at /var/log/helk-install.log if you are having install errors

Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 DEP-11 Metadata [51.4 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2,212 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 DEP-11 Metadata [57.9 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 DEP-11 Metadata [2,464 B]
Get:9 http://us.archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,342 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 DEP-11 Metadata [293 kB]
Get:11 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,749 kB]
Get:12 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,575 kB]
Get:13 http://us.archive.ubuntu.com/ubuntu bionic-updates/universe amd64 DEP-11 Metadata [295 kB]
Get:14 http://us.archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 DEP-11 Metadata [2,468 B]
Get:15 http://us.archive.ubuntu.com/ubuntu bionic-backports/universe amd64 DEP-11 Metadata [9,272 B]
Fetched 7,841 kB in 5s (1,647 kB/s)
Reading package lists...

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  python3-click python3-colorama
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libapr1 libaprutil1
The following NEW packages will be installed:
  apache2-utils libapr1 libaprutil1
0 upgraded, 3 newly installed, 0 to remove and 1 not upgraded.
Need to get 259 kB of archives.
After this operation, 866 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 libapr1 amd64 1.6.3-2 [90.9 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic/main amd64 libaprutil1 amd64 1.6.1-2 [84.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 apache2-utils amd64 2.4.29-1ubuntu4.16 [84.0 kB]
Fetched 259 kB in 2s (129 kB/s)
Selecting previously unselected package libapr1:amd64.
(Reading database ... 158537 files and directories currently installed.)
Preparing to unpack .../libapr1_1.6.3-2_amd64.deb ...
Unpacking libapr1:amd64 (1.6.3-2) ...
Selecting previously unselected package libaprutil1:amd64.
Preparing to unpack .../libaprutil1_1.6.1-2_amd64.deb ...
Unpacking libaprutil1:amd64 (1.6.1-2) ...
Selecting previously unselected package apache2-utils.
Preparing to unpack .../apache2-utils_2.4.29-1ubuntu4.16_amd64.deb ...
Unpacking apache2-utils (2.4.29-1ubuntu4.16) ...
Setting up libapr1:amd64 (1.6.3-2) ...
Setting up libaprutil1:amd64 (1.6.1-2) ...
Setting up apache2-utils (2.4.29-1ubuntu4.16) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
Adding password for user helk

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
  python3-click python3-colorama
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libcurl4
The following NEW packages will be installed:
  curl libcurl4
0 upgraded, 2 newly installed, 0 to remove and 1 not upgraded.
Need to get 378 kB of archives.
After this operation, 1,051 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 libcurl4 amd64 7.58.0-2ubuntu3.14 [219 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu bionic-updates/main amd64 curl amd64 7.58.0-2ubuntu3.14 [159 kB]
Fetched 378 kB in 2s (155 kB/s)
Selecting previously unselected package libcurl4:amd64.
(Reading database ... 158586 files and directories currently installed.)
Preparing to unpack .../libcurl4_7.58.0-2ubuntu3.14_amd64.deb ...
Unpacking libcurl4:amd64 (7.58.0-2ubuntu3.14) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.58.0-2ubuntu3.14_amd64.deb ...
Unpacking curl (7.58.0-2ubuntu3.14) ...
Setting up libcurl4:amd64 (7.58.0-2ubuntu3.14) ...
Setting up curl (7.58.0-2ubuntu3.14) ...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Processing triggers for libc-bin (2.27-3ubuntu1.4) ...
# Executing docker install script, commit: 93d2499759296ac1f9c510605fef85052a2c32be
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | gpg --dearmor --yes -o /usr/share/keyrings/docker-archive-keyring.gpg
gpg: WARNING: unsafe ownership on homedir '/home/user1/.gnupg'
+ sh -c echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu bionic stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends  docker-ce-cli docker-scan-plugin docker-ce >/dev/null
+ version_gte 20.10
+ [ -z  ]
+ return 0
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq docker-ce-rootless-extras >/dev/null
+ sh -c docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:08 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:16 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   633  100   633    0     0   1715      0 --:--:-- --:--:-- --:--:--  1710
100 11.6M  100 11.6M    0     0  9756k      0  0:00:01  0:00:01 --:--:-- 9756k
Creating network "docker_helk" with driver "bridge"
Creating volume "docker_esdata" with local driver
Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:7.6.2)...
7.6.2: Pulling from elasticsearch/elasticsearch
Digest: sha256:59342c577e2b7082b819654d119f42514ddf47f0699c8b54dc1f0150250ce7aa
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.6.2
Pulling helk-kibana (docker.elastic.co/kibana/kibana:7.6.2)...
7.6.2: Pulling from kibana/kibana
Digest: sha256:e8f3743e404462709663422056db2d5076a7a6bd6024f64aea1599b3014c63be
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.6.2
Pulling helk-logstash (otrf/helk-logstash:7.6.2.1)...
7.6.2.1: Pulling from otrf/helk-logstash
Digest: sha256:b1135da506f40fc1d5861db7ba844486f3a08a57af3fdb8e301ab487f51a2ac1
Status: Downloaded newer image for otrf/helk-logstash:7.6.2.1
Pulling helk-nginx (otrf/helk-nginx:0.3.0)...
0.3.0: Pulling from otrf/helk-nginx
Digest: sha256:32eb6e39681849dc3bed36cfb95bd39b25f8c66d08965b6855f64eb2ee0668ba
Status: Downloaded newer image for otrf/helk-nginx:0.3.0
Pulling helk-zookeeper (otrf/helk-zookeeper:2.4.0)...
2.4.0: Pulling from otrf/helk-zookeeper
Digest: sha256:d8a7c57c03384f5ce2b6125505c1f8e2a020432de81bde3677fcc8009fc5cfd2
Status: Downloaded newer image for otrf/helk-zookeeper:2.4.0
Pulling helk-kafka-broker (otrf/helk-kafka-broker:2.4.0)...
2.4.0: Pulling from otrf/helk-kafka-broker
Digest: sha256:22b87b2e2c97157471af3db8a19e85c9184fa492fa8cd67cc57617c6abec6dce
Status: Downloaded newer image for otrf/helk-kafka-broker:2.4.0
Pulling helk-ksql-server (confluentinc/ksqldb-server:latest)...
latest: Pulling from confluentinc/ksqldb-server
Digest: sha256:a75f49a54d287356337f64dcf81d9ce8a8e1932e999904568b1abd867b3ca7c7
Status: Downloaded newer image for confluentinc/ksqldb-server:latest
Pulling helk-ksql-cli (confluentinc/ksqldb-cli:latest)...
latest: Pulling from confluentinc/ksqldb-cli
Digest: sha256:a75f49a54d287356337f64dcf81d9ce8a8e1932e999904568b1abd867b3ca7c7
Status: Downloaded newer image for confluentinc/ksqldb-cli:latest
Creating helk-elasticsearch ... done
Creating helk-kibana        ... done
Creating helk-nginx         ... done
Creating helk-logstash      ... done
Creating helk-zookeeper     ... done
Creating helk-kafka-broker  ... done
Creating helk-ksql-server   ... done
Creating helk-ksql-cli      ... done

What version of HELK are you using

run the command from within the HELK repo run git log -1 --oneline

ad752b2 (HEAD -> master, origin/master, origin/HEAD) Update jvm.options (#563)

What version of Winlogbeat are you using if you are using Windows/WEF logs

 NA as importing mordor logs into HELK using kcat
What steps did you take trying to fix the issue
  • jq shows no error for the test jsons, either mordor sample or my own (can share, if needed)
  • I created a sample which had a format { "events": [ {evt1},\n{evt2},\n{evt3},\n ....]}. The key:value pairs in this sample are separated by NL/CR. Upon importing this sample using kcat, the HELK discovery page shows _jsonParse_failure and no data from the imported logs shows up here. Global dashboard for this sample shows number of lines in the sample log file under global_count widget & the rest widgets, dashboards remained unpopulated.
  • Modified my sample logs to match mordor_project log format, i.e 1 line / event_log ex: {evt1}\n{evt2}\n{evt3}\n ..... This did not even got ingested (sample logs can be shared)
How could we replicate the issue
  • install HELK on ubuntu 18.04, choose option 1
  • import https://github.com/OTRF/Security-Datasets/blob/master/datasets/atomic/windows/lateral_movement/host/empire_wmic_add_user_backdoor.zip using kcat (winlogbeat as type)
Any additionally code or log context you would like to provide

Please see attached screen shots
Let me know, if you want the sample sysmon logs I tested with.

Any additional context or input you have

pictures, comments, etc.
W_all-mitre
W_mitre_groups
N_sysmon
N_global
N_process
incorrect_timestamp 2021-09-12 at 12 30 47 AM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@splunk-user1