Malcolm v24.01.0

[mmguero]

Malcolm v24.01.0 contains new features, improvements, bug fixes and component version updates.

v23.12.1...v24.01.0

• Features and enhancements
  • new Malcolm instance landing page (Malcolm instance landing page idaholab/Malcolm#252)
  • file carve download with password-protected .zip file (file carve download with password-protected .zip file idaholab/Malcolm#288)
  • new "all files exept common plain text files" option for Malcolm's file carving to match Hedgehog capability (add "all files exept common plain text files" to malcolm's carving like hedgehog has idaholab/Malcolm#290)
  • allow customizing indexes for logs written to OpenSearch/Elasticsearch (allow customizing indexes for logs written to OpenSearch/Elasticsearch idaholab/Malcolm#313)
  • more consistently differentiate between uploaded and live-captured traffic (more consistently differentiate between uploaded and live-captured traffic idaholab/Malcolm#321)
  • make download extracted file context item from Arkime smarter (make download extracted file context item from Arkime smarter idaholab/Malcolm#330)
  • improve netbox device type library import by using "official" import script (improve netbox device type library import by using "official" import script idaholab/Malcolm#384)
• Component version updates
  • Alpine Linux to v3.19 as the base for some Docker images
  • Fluent Bit to v2.2.2
  • Beats to v8.11.4
  • LogStash to v8.11.4
• Bug fixes
  • Suricata Alerts dashboard "Alerts - Tags" visualization is useless (Suricata Alerts dashboard "Alerts - Tags" visualization is useless idaholab/Malcolm#314)
  • third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm (third party logs are not parsed correctly from fluentbit -> fluentd aggregator -> Malcolm idaholab/Malcolm#318)
  • update document lookup APIs to search either network or host data (update document lookup APIs to search either network or host data idaholab/Malcolm#322)
  • suricata rule update is broken (suricata rule update is broken idaholab/Malcolm#323)
  • time sync from hedgehog to Malcolm opensearch instance not working (time sync from hedgehog to Malcolm opensearch instance not working idaholab/Malcolm#324)
  • fix issue specifying database mode via command-line
  • have pruning of OpenSearch indices (based on size) include "other" Malcolm indices as well (e.g., nginx logs, system resources, third-party logs, etc.)
• Configuration changes (in environment variables in ./config/)
  • added the following variables with relation to allow customizing indexes for logs written to OpenSearch/Elasticsearch idaholab/Malcolm#313
    • added ARKIME_ROTATE_INDEX to arkime.env with default value of daily (see Arkime docs on rotateIndex)
    • added the following variables and defaults to opensearch.env:

    # OpenSearch index patterns and timestamp fields
    # Index pattern for network traffic logs written via Logstash (e.g., Zeek logs, Suricata alerts)
    MALCOLM_NETWORK_INDEX_PATTERN=arkime_sessions3-*
    # Default time field to use for network traffic logs in Logstash and Dashboards
    MALCOLM_NETWORK_INDEX_TIME_FIELD=firstPacket
    # Suffix used to create index to which network traffic logs are written (supports Ruby strftime strings in %{})
    MALCOLM_NETWORK_INDEX_SUFFIX=%{%y%m%d}
    # Index pattern for other logs written via Logstash (e.g., nginx, beats, fluent-bit, etc.)
    MALCOLM_OTHER_INDEX_PATTERN=malcolm_beats_*
    # Default time field to use for other logs in Logstash and Dashboards
    MALCOLM_OTHER_INDEX_TIME_FIELD=@timestamp
    # Suffix used to create index to which other logs are written (supports Ruby strftime strings in %{})
    MALCOLM_OTHER_INDEX_SUFFIX=%{%y%m%d}
    # Index pattern used specifically by Arkime (will probably match MALCOLM_NETWORK_INDEX_PATTERN, should probably be arkime_sessions3-*)
    ARKIME_NETWORK_INDEX_PATTERN=arkime_sessions3-*
    # Default time field used by for sessions in Arkime viewer
    ARKIME_NETWORK_INDEX_TIME_FIELD=firstPacket
    

  • changed default for EXTRACTED_FILE_HTTP_SERVER_KEY to infected in zeek-secret.env
  • added EXTRACTED_FILE_HTTP_SERVER_ZIP with default value of false in zeek.env, see (file carve download with password-protected .zip file idaholab/Malcolm#288)