malcolm v1.5 development (#56)

* bump versions of Zeek and Moloch and Elastic/beats
* added HASSH fingerprinting for SSH
* added Community ID fingerprinting for flows
* detect and upgrade Moloch administrative tables on startup
* default to faster java execution engine
* lots of bug fixes and tweaks

-----------------------------------
individual commit comments:

* tweaks to build of malcolm iso and log filtering

* sync sensor shared script with malcolm shared script

* Bump Zeek version to 2.6.3

* reduce log verbosity

* tweaks to build of malcolm iso and log filtering

* tweak config file for remote sensor communication

* bump version to 1.4.1

* sync sensor shared script with malcolm shared script

* Restore syslinux entry for ISO build

* ISO/Docker build improvements:

1. Make sure the ISO is built with the right version number in the name
2. Spot check the contents of the docker images to make sure files were built/pulled correctly

* ISO build tweaks

* bump elastic version

* added query syntax cheat sheet

* bump development version to 1.5.0

* for issue #42, enable community_id for zeek plugins to populate conn.log

* issue #44: cast newly inputted values for settings.json to integers before storing

* Development for Malcolm v1.5 (#49)

* update moloch to 2.0, issue #46

* Disable detect-MHR.bro by default in local.bro.

As we don't want Zeek reaching out to the internet by default to do hash lookups, disable policy/frameworks/files/detect-MHR.bro in local.bro.

See https://docs.zeek.org/en/stable/scripts/policy/frameworks/files/detect-MHR.bro.html
See https://www.team-cymru.com/mhr.html#DNS

In the meantime people who want this can override it with a volume mount in docker with the value uncommented. I may put it back in with an environment variable to enable it if desired.

* add debugging code to zeek-carve-monitor.py

* Development for Moloch v1.5 (#50)

* update moloch to 2.0, issue #46

* Disable detect-MHR.bro by default in local.bro.

As we don't want Zeek reaching out to the internet by default to do hash lookups, disable policy/frameworks/files/detect-MHR.bro in local.bro.

See https://docs.zeek.org/en/stable/scripts/policy/frameworks/files/detect-MHR.bro.html
See https://www.team-cymru.com/mhr.html#DNS

In the meantime people who want this can override it with a volume mount in docker with the value uncommented. I may put it back in with an environment variable to enable it if desired.

* add debugging code to zeek-carve-monitor.py

* don't rely on environment variable INITIALIZEDB in docker-compose.yml to determine if Moloch has created the Elasticsearch database configuration files. Instead look and see if zeek_template has been loaded or not

fixes issue 45: INITIALIZEDB environment variable and restart value in docker-compose.yml could cause moloch container to wipe elasticsearch database on every reboot

* Create startup.nsh for EFI boot entry grubx64.efi; fixes issue with VirtualBox/VMWare not being able to find the EFI boot entry. Should not affect hardware boot.

* Development for Malcolm v1.5 (#52)

* work on issue #47, handle moloch db.pl upgrades

When moloch changes its index definition, it needs to be upgraded with db.pl

Upon starting Malcolm's Moloch docker container I am now comparing the list of all of the indexes against the known current versions of the administrative indices. If they don't match, I run the upgrade script before starting capture/viewer.

see https://molo.ch/faq#upgrading-moloch
see https://molo.ch/faq#how_do_i_upgrade_to_moloch_2

* improve tagging and views for routable (public) IPv4 addresses; issue #51

* fixed table formatting

* Development: sync scripts from hedgehog (#54)

* sync scripts from hedgehog sensor

* sync scripts from hedgehog sensor

* default LOGSTASH_JAVA_EXECUTION_ENGINE to true for better logstash performance

* tweak local.bro, enable credential gathering

* Issue #55, include HASSH in Zeek plugins for SSH fingerprinting

* Added database of JA3 fingerprints from https://ja3er.com

* fix session sort direction for new users

* add documentation for community ID/zeek uid correlation

* added patch to fix incorrect quoting, cherry-picked from aol/moloch:master commit 4de1686 for moloch issue 1146

Dockerfiles/filebeat.Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/beats/filebeat-oss:6.8.1
+FROM docker.elastic.co/beats/filebeat-oss:6.8.2
 
 # Copyright (c) 2019 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"

Dockerfiles/kibana.Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.elastic.co/kibana/kibana-oss:6.8.1
+FROM docker.elastic.co/kibana/kibana-oss:6.8.2
 
 # Copyright (c) 2019 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
@@ -40,9 +40,9 @@ ADD kibana/supervisord.conf /etc/supervisord.conf
 ADD kibana/dashboards /opt/kibana/dashboards
 ADD kibana/maps /opt/maps
 ADD https://github.com/bitsensor/elastalert-kibana-plugin/releases/download/1.0.3/elastalert-kibana-plugin-1.0.3-6.8.0.zip /tmp/elastalert-kibana-plugin.zip
-ADD https://github.com/prelert/kibana-swimlane-vis/releases/download/v6.7.1/prelert_swimlane_vis-6.7.1.zip /tmp/kibana-swimlane.zip
+ADD https://github.com/prelert/kibana-swimlane-vis/releases/download/v6.8.1/prelert_swimlane_vis-6.8.1.zip /tmp/kibana-swimlane.zip
 ADD https://github.com/gwintzer/kibana-comments-app-plugin/releases/download/6.7.1/kibana-comments-app-plugin-6.7.1-1.zip /tmp/kibana-comments.zip
-ADD https://github.com/walterra/kibana-milestones-vis/releases/download/v6.8.0/kibana-milestones-vis-6.8.0.zip /tmp/kibana-milestones.zip
+ADD https://github.com/walterra/kibana-milestones-vis/releases/download/v6.8.1/kibana-milestones-vis-6.8.1.zip /tmp/kibana-milestones.zip
 
 # TODO: commented out because it's not optimizing in 6.6+ correctly
 # put these back in here and below in the build section:
@@ -63,25 +63,25 @@ RUN chmod 755 /data/*.sh /data/*.py && \
     cd /tmp && \
     echo "Installing ElastAlert plugin..." && \
       unzip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json && \
-      sed -i "s/6\.8\.0/6\.8\.1/g" kibana/elastalert-kibana-plugin/package.json && \
+      sed -i "s/6\.8\.0/6\.8\.2/g" kibana/elastalert-kibana-plugin/package.json && \
       zip elastalert-kibana-plugin.zip kibana/elastalert-kibana-plugin/package.json && \
       /usr/share/kibana/bin/kibana-plugin install file:///tmp/elastalert-kibana-plugin.zip && \
       rm -f /tmp/elastalert-kibana-plugin.zip && \
     echo "Installing Swimlanes visualization..." && \
-      unzip kibana-swimlane.zip kibana/prelert_swimlane_vis-6.7.1/package.json && \
-      sed -i "s/6\.7\.1/6\.8\.1/g" kibana/prelert_swimlane_vis-6.7.1/package.json && \
-      zip kibana-swimlane.zip kibana/prelert_swimlane_vis-6.7.1/package.json && \
+      unzip kibana-swimlane.zip kibana/prelert_swimlane_vis-6.8.1/package.json && \
+      sed -i "s/6\.8\.1/6\.8\.2/g" kibana/prelert_swimlane_vis-6.8.1/package.json && \
+      zip kibana-swimlane.zip kibana/prelert_swimlane_vis-6.8.1/package.json && \
       /usr/share/kibana/bin/kibana-plugin install file:///tmp/kibana-swimlane.zip && \
       rm -f /tmp/elastalert-kibana-plugin.zip && \
     echo "Installing Comments visualization..." && \
       unzip kibana-comments.zip kibana/kibana-comments-app-plugin/package.json && \
-      sed -i "s/6\.7\.1/6\.8\.1/g" kibana/kibana-comments-app-plugin/package.json && \
+      sed -i "s/6\.7\.1/6\.8\.2/g" kibana/kibana-comments-app-plugin/package.json && \
       zip kibana-comments.zip kibana/kibana-comments-app-plugin/package.json && \
       /usr/share/kibana/bin/kibana-plugin install file:///tmp/kibana-comments.zip && \
       rm -rf /tmp/kibana-comments.zip /tmp/kibana && \
     echo "Installing Milestones visualization..." && \
       unzip kibana-milestones.zip kibana/kibana-milestones-vis/package.json && \
-      sed -i "s/6\.8\.0/6\.8\.1/g" kibana/kibana-milestones-vis/package.json && \
+      sed -i "s/6\.8\.1/6\.8\.2/g" kibana/kibana-milestones-vis/package.json && \
       zip kibana-milestones.zip kibana/kibana-milestones-vis/package.json && \
       /usr/share/kibana/bin/kibana-plugin install file:///tmp/kibana-milestones.zip && \
       rm -rf /tmp/kibana-milestones.zip /tmp/kibana

Dockerfiles/logstash.Dockerfile

@@ -3,7 +3,7 @@ FROM centos:7 AS build
 # Copyright (c) 2019 Battelle Energy Alliance, LLC.  All rights reserved.
 LABEL maintainer="[email protected]"
 
-ARG LOGSTASH_JAVA_EXECUTION_ENGINE=false
+ARG LOGSTASH_JAVA_EXECUTION_ENGINE=true
 
 ENV LOGSTASH_JAVA_EXECUTION_ENGINE $LOGSTASH_JAVA_EXECUTION_ENGINE
 
@@ -24,7 +24,7 @@ RUN /bin/bash -lc "command curl -sSL https://rvm.io/mpapis.asc | gpg2 --import -
     git clone --depth 1 https://github.com/mmguero/logstash-filter-ieee_oui.git /opt/logstash-filter-ieee_oui && \
     /bin/bash -lc "cd /opt/logstash-filter-ieee_oui && bundle install && gem build logstash-filter-ieee_oui.gemspec && bundle info logstash-filter-ieee_oui"
 
-FROM docker.elastic.co/logstash/logstash-oss:6.8.1 AS runtime
+FROM docker.elastic.co/logstash/logstash-oss:6.8.2 AS runtime
 
 USER root
 

Dockerfiles/moloch.Dockerfile

@@ -5,11 +5,12 @@ LABEL maintainer="[email protected]"
 
 ENV DEBIAN_FRONTEND noninteractive
 
-ENV MOLOCH_VERSION "1.8.0"
+ENV MOLOCH_VERSION "2.0.0"
 ENV MOLOCHDIR "/data/moloch"
-ENV ZEEK_VERSION "2.6.2"
+ENV ZEEK_VERSION "2.6.4"
 ENV ZEEK_DIR "/opt/bro"
 ENV CYBERCHEF_VERSION "8.30.1"
+ENV ZEEK_CORELIGHT_COMMUNITY_ID_PLUGIN_VER "1.2"
 
 ADD moloch/scripts/bs4_remove_div.py /data/
 ADD moloch/patch/* /data/patches/
@@ -19,8 +20,9 @@ ADD docs/images $MOLOCHDIR/doc/images/
 ADD https://github.com/aol/moloch/archive/v$MOLOCH_VERSION.tar.gz /data/moloch.tar.gz
 ADD https://github.com/gchq/CyberChef/releases/download/v$CYBERCHEF_VERSION/cyberchef.htm $MOLOCHDIR/doc/cyberchef.htm
 ADD https://www.zeek.org/downloads/bro-$ZEEK_VERSION.tar.gz /data/bro.tar.gz
+ADD https://github.com/corelight/bro-community-id/archive/$ZEEK_CORELIGHT_COMMUNITY_ID_PLUGIN_VER.tar.gz /data/bro-community-id.tar.gz
 
-RUN sed -i "s/stretch main/stretch main contrib non-free/" /etc/apt/sources.list && \
+RUN sed -i "s/stretch main/stretch main contrib non-free/g" /etc/apt/sources.list && \
     apt-get -q update && \
     apt-get install -q -y --no-install-recommends \
         bison \
@@ -77,6 +79,16 @@ RUN sed -i "s/stretch main/stretch main contrib non-free/" /etc/apt/sources.list
     mkdir -p $ZEEK_DIR/share/bro/site/ja3 && \
     cp -v /tmp/ja3/bro/* $ZEEK_DIR/share/bro/site/ja3 && \
     rm -rf /tmp/ja3 && \
+  git clone --depth 1 https://github.com/salesforce/hassh /tmp/hassh && \
+    mkdir -p $ZEEK_DIR/share/bro/site/hassh && \
+    cp -v /tmp/hassh/bro/* $ZEEK_DIR/share/bro/site/hassh && \
+    rm -rf /tmp/hassh && \
+  cd /data && \
+    tar -xvf "bro-community-id.tar.gz" && \
+    cd "bro-community-id-"$ZEEK_CORELIGHT_COMMUNITY_ID_PLUGIN_VER && \
+    ./configure --bro-dist="/data/bro-"$ZEEK_VERSION --install-root=$ZEEK_DIR/lib/bro/plugins && \
+    make && \
+    make install && \
   cd $MOLOCHDIR/doc/images && \
     find . -name "*.png" -exec bash -c 'convert "{}" -fuzz 2% -transparent white -background white -alpha remove -strip -interlace Plane -quality 85% "{}.jpg" && rename "s/\.png//" "{}.jpg"' \; && \
     cd $MOLOCHDIR/doc && \
@@ -109,7 +121,14 @@ RUN sed -i "s/stretch main/stretch main contrib non-free/" /etc/apt/sources.list
     ./easybutton-build.sh --install && \
     npm cache clean --force && \
   apt-get clean && \
-  rm -rf $MOLOCHDIR"-"$MOLOCH_VERSION "/data/bro-"$ZEEK_VERSION /var/lib/apt/lists/* /tmp/* /var/tmp/*
+  rm -rf $MOLOCHDIR"-"$MOLOCH_VERSION \
+         /data/bro.tar.gz \
+         "/data/bro-"$ZEEK_VERSION \
+         /data/bro-community-id.tar.gz \
+         "/data/bro-community-id-"$ZEEK_CORELIGHT_COMMUNITY_ID_PLUGIN_VER \
+         /var/lib/apt/lists/* \
+         /tmp/* \
+         /var/tmp/*
 
 FROM debian:stretch-slim AS runtime
 
@@ -126,10 +145,6 @@ ARG MOLOCH_ANALYZE_PCAP_THREADS=1
 ARG CAPTURE=off
 ARG WISE=off
 ARG VIEWER=on
-#Initalize is used to reset the environment from scratch and rebuild a new ES Stack
-ARG INITIALIZEDB=false
-#Wipe is the same as initalize except it keeps users intact
-ARG WIPEDB=false
 #Whether or not Moloch is in charge of deleting old PCAP files to reclaim space
 ARG MANAGE_PCAP_FILES=false
 #Whether or not to auto-tag logs based on filename
@@ -156,8 +171,6 @@ ENV MOLOCH_ANALYZE_PCAP_THREADS $MOLOCH_ANALYZE_PCAP_THREADS
 ENV CAPTURE $CAPTURE
 ENV WISE $WISE
 ENV VIEWER $VIEWER
-ENV INITIALIZEDB $INITIALIZEDB
-ENV WIPEDB $WIPEDB
 ENV MANAGE_PCAP_FILES $MANAGE_PCAP_FILES
 ENV AUTO_TAG $AUTO_TAG
 ENV ZEEK_DIR "/opt/bro"
@@ -239,8 +252,6 @@ RUN groupadd --gid 1000 $MOLOCHUSER && \
 #Update Path
 ENV PATH="/data:$MOLOCHDIR/bin:$ZEEK_DIR/bin:${PATH}"
 
-VOLUME ["/data/configured"]
-
 EXPOSE 8000 8005 8081
 WORKDIR $MOLOCHDIR