Malcolm v5.2.3

Malcolm v5.2.3 is a patch release with component version bumps, bug fixes and improvements.

v5.2.2...v5.2.3

• Version bumps

  • Arkime v3.3.1
  • Zeek v4.2.0

• Improvements

  • Added script and better documentation for putting Malcolm in "read-only" mode
  • Improved Files dashboard

• Bug fixes

  • Fixed an issue where Logstash wasn't parsing the ftime from files.log correctly (a field added by the Spicy ZIP analyzer)
  • Fixed idaholab#73 (path for tcpdump changed) for Hedgehog Linux
  • Fixed idaholab#72 (better file directory/name parsing and normalization in Logstash)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.2

Malcolm v5.2.2 is a patch release with some improvements to the API and a fix for using Zeek intelligence files on Hedgehog Linux.

v5.2.1...v5.2.2

• Added more capabilities to the API
  • added /document/ API
  • added filter ability to /agg/ and /document/ API
  • added more documentation and examples
• For Zeek intel. files, changed location from /opt/zeek/share/zeek/site/intel to /opt/sensor/sensor_ctl/zeek/intel so that they aren't lost on reboot

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.1

Malcolm v5.2.1 is patch release identical to v5.2.0 with the addition of a fix (arkime/arkime@f13e936) for a regression bug introduced in Arkime v3.3.0 which prevented the Arkime viewer from correctly loading some large or XORed packets.

In addition, a minor change was made to the startup scripts for Hedgehog Linux's Zeek configuration to allow Zeek intelligence files to be automatically loaded the same way they are in Malcolm's Zeek container.

v5.2.0...v5.2.1

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.2.0

Malcolm v5.2.0 is a feature release with a several new features and improvements, version bumps and bug fixes.

EDIT: As of this morning (1/21/2022) I'm tracking a regression in Arkime v3.3.0 with viewing the packet payload of some large sessions. It's likely a patch release will be put out later today to address this. Apologies.

v5.1.0...v5.2.0

• New features

  • Zeek Intelligence Framework (see idaholab#20)
    • To quote Zeek's Intelligence Framework documentation, "The goals of Zeek’s Intelligence Framework are to consume intelligence data, make it available for matching, and provide infrastructure to improve performance and memory utilization. Data in the Intelligence Framework is an atomic piece of intelligence such as an IP address or an e-mail address. This atomic data will be packed with metadata such as a freeform source field, a freeform descriptive field, and a URL which might lead to more information about the specific item." Zeek intelligence indicator types include IP addresses, URLs, file names, hashes, email addresses, and more.
    • Malcolm doesn't come bundled with intelligence files from any particular feed, but they can be easily included into your local instance. On startup, Malcolm's malcolmnetsec/zeek docker container enumerates the subdirectories under ./zeek/intel (which is bind mounted into the container's runtime) and configures Zeek so that those intelligence files will be automatically included in its local policy. Subdirectories under ./zeek/intel which contain their own __load__.zeek file will be @load-ed as-is, while subdirectories containing "loose" intelligence files will be loaded automatically with a redef Intel::read_files directive.
  • New OPCUA Binary protocol parser for Zeek and corresponding dashboard.

• Improvements

  • set ecs.provider to arkime for logs from Arkime's capture to make categorizing logs by source easier
  • API
    • allow bucketing multiple fields from /agg/ API
    • added /fields/ API to list fields
      added documentation
  • ECS normalization to related.hosts field for all applicable protocols
  • updated documentation, screenshots and slides
  • spreadsheet mapping STIX v1.2 fields to Zeek fields and Malcolm normalized fields
  • updated MITRE ATT&CK mappings for Capa hits
  • added a pseudo-read-only NGINX configuration

• Version bumps

  • Arkime to v3.3.0
  • OpenSearch to v1.2.4
  • Capa to v3.1.0
  • cve-2021-44228 Log4Shell detector plugin for Zeek to v0.5.3 (see corelight/cve-2021-44228#46)

• Bug Fixes

  • fix idaholab#71 (type mismatch for network.vlan.id between Malcolm and Arkime definitions) which prevented vlan traffic from indexing correctly from Arkime's capture with Malcolm's field template
  • fix for ethernet/IP traffic which could lead to Zeek runaway memory allocation until crash: "Fixed bug with Request Paths containing Port Segments" (cisagov/icsnpp-enip@4696a43)

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.1.0

Malcolm v5.1.0 is a feature release laying the groundwork for a new REST API for querying Malcolm. It also contains a few component version bumps.

v5.0.4...v5.1.0

• New features
  • put framework in for Malcolm REST API (idaholab#70) - not feature complete yet, but minimally usable
• Version bumps
  • OpenSearch to v1.2.3
  • LogStash Docker image to v7.16.2 with OpenSearch output plugin v1.2.0
  • Latest releases of Zeek packages
• Misc.
  • Reformatted all Python code with Black with the options --line-length 120 --skip-string-normalization
  • Updated some deprecated logstash filter parameters in translate filter

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.4

Malcolm v5.0.4 is a patch release with improvements to Zeek detection of CVE-2021-44228 ("Log4Shell" Log4J vulnerability).

v5.0.3...v5.0.4

• build with latest corelight/cve-2021-44228 release

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.3

Malcolm v5.0.3 is a patch release with a few minor bug fixes and improvements to Zeek detection of CVE-2021-44228 ("Log4Shell" Log4J vulnerability).

v5.0.2...v5.0.3

• build with latest zeek/spicy-ldap release (dpd-based detection rather than just port-based)
• build with latest corelight/cve-2021-44228 release
• fix idaholab#69 (zeek resists shutdown on sensor during halt/reboot)
• bump OpenSearch to v1.2.2 which has log4j 2.16
• added convenience script for working with GitHub workflow-built images

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.2

Malcolm v5.0.2 is a patch release adding HTTP header-based Zeek detection of CVE-2021-44228 ("Log4Shell" Log4J vulnerability).

v5.0.1...v5.0.2

• Added Corelight's Zeek detection script for CVE-2021-44228 ("Log4Shell" Log4J vulnerability)
• move zeek.http.tags field up to top-level tags
• Version bumps
  • Arkime to v3.2.1
  • Alpine (for dashboards-helper, name-map-ui and nginx-proxy Docker containers) to v3.15.0
  • NGINX (for nginx-proxy Docker container) to v1.20.2

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.1

Malcolm v5.0.1 is a patch release with minor bug- and security-related fixes.

v5.0.0...v5.0.1

• Security vulnerabilities addressed:

  • mitigations for CVE-2021-44228 (log4shell) idaholab#68

• Bugs addressed:

  • Very large pcaps don't get proccesed idaholab#44
  • pcap files with colon (:) in the name don't process correctly idaholab#2
  • turning off AUTO_TAG feature disables tagging altogether idaholab#12
  • recent debinterfaces release broke configure-interfaces.py idaholab#48
  • opensearch indexes in yellow state idaholab#67
  • arkime capture gives mlockall_init() warning on startup idaholab#66

• Other

  • bumped Arkime from v3.1.1 to v3.2.0
  • bumped OpenSearch to v1.2.1
  • switched from elasticsearch to opensearch python client libraries
  • write contributor's guide for source code contributions/modifications idaholab#25
  • handle new fields in ethernet/IP logs (cisagov/icsnpp-enip@c4ae505)
  • use more recognizable dashboards logo for OpenSearch dashboards launcher in Malcolm ISO
  • include patches used to build Arkime Dockerfile when building Arkime for hedgehog as well
  • build Zeek spicy analyzers from their various repos rather than the zeek/spicy-analyzer meta-repo

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Malcolm v5.0.0

Malcolm v5.0.0 is a major release which addresses idaholab#54, transition from ElasticSearch to OpenSearch

v4.0.1...v5.0.0

Malcolm has switched to the OpenSearch project as the basis of its search and analytics capabilities, mainly for two reasons:

1. Elastic.co's decision to no longer release Elasticsearch and Kibana under an open source license
2. Capabilities available under OpenSearch (and previously under Open Distro for Elasticsearch) that are only available with paid "premium" Elastic.co subscriptions (machine learning anomaly detection, alerting, reporting, etc.)

As the Malcolm project uses semantic versioning when choosing version numbers, this backwards-compatibility breaking change is the reason for bumping the major version number from 4 to 5. It is not recommended to attempt an upgrade from a previous release; a fresh install is required.

Malcolm and Hedgehog Linux may be obtained by pulling or building the Docker images and/or building the ISO installer images as described in the documentation. Unofficial ISO installer images for Malcolm and Hedgehog Linux are not hosted on GitHub, but may be downloaded from https://malcolm.fyi/download/.

Historical context for the events and reasoning behind this change:

• Elastic announces license change

  • Amazon NOT OK
  • Doubling Down on Open
  • Doubling Down on Open, Part II
  • Elastic License v2
  • FAQ on 2021 License Change
    • Does this mean that Elasticsearch and Kibana are no longer Open Source? Yes. Neither the Elastic License nor SSPL have been approved by the OSI, so to prevent confusion, we no longer refer to Elasticsearch or Kibana as open source.
  • old "open source" tier ("Apache 2.0: Now and always" 🙄) goes away
  • The SSPL is not an open source license

• OpenSearch fork:

  • Stepping up for a truly open source Elasticsearch
  • Introducing OpenSearch
  • Truly Doubling Down on Open Source @ logz.io
  • FAQ

• Third-party blogs, etc.

  • Elasticsearch does not belong to Elastic
  • Elasticearch and Kibana are now business risks
  • Is Elasticsearch no longer open source software?
  • The Implications of Elasticsearch and Kibana License Change
  • Let's talk about the Elastic license change
  • Elastic is going closed-source. Where does that leave MSSPs?