Malcolm v25.02.0 (see NOTE prior to upgrade)

[mmguero]

Malcolm v25.02.0 contains some major performance improvements, a few smaller new features and enhancements, several component version updates, bug fixes, and documentation updates.

v25.01.0...v25.02.0

NOTE: As a result of some of the changes to environment variables made for decoupling Redis from NetBox (#580), environment variables from previous version may cause NetBox to fail to connect to Redis which prevents successful startup. To fix this, you should perform the following steps once prior to starting Malcolm:

1. Stop Malcolm (./scripts/stop)
2. Change the values for REDIS_CACHE_HOST and REDIS_HOST, removing the netbox- prefix from the values, so that they look like REDIS_HOST=redis and REDIS_CACHE_HOST=redis-cache, respectively.
   • These values were found in netbox.env in previous versions, but are found in redis.env in this release.
   • Alternatively, you may remove the lines for REDIS_HOST and REDIS_CACHE_HOST completely and they will be restored with correct values the next time the control script is run.
3. Run ./scripts/status which will check the .env files and restore the missing values if you removed them.
4. Start Malcolm (./scripts/start)

• ✨ Features and enhancements
  • performance improvements for NetBox enrichment (#547)
    • NetBox enrichment and autopopulation is now approximately 4x faster than it was before (depending on resources)
  • performance improvements for Suricata's processing of uploaded PCAP files (#457)
    • Suricata's processing of large sets of uploaded PCAP files is now approximately 18x faster faster than it was before (depending on resources)
  • add validate_local_site_policy.sh script for validating Zeek local site policy (#598)
  • include corelight/zeek-long-connections plugin to log long connections (#585)
    • new zeek.conn.long field is available to indicate long connections
    • Connections dashboard updated to include this new field
    • see notes below on environment variable additions for configuring this plugin
  • standardize container health checks into scripts for all containers (#491)
    • added container health checks for containers that did not previously have them (live capture containers)
  • significant work-in-progress towards support for Sigma rules via OpenSearch Security Analytics (still incomplete due to some blocking issues upstream, see #475 for details)
    • changed normalization of Windows event log records (evtx) to more closely match Winlogbeat fields which are closer to what the Sigma rules for Windows events use, and updated corresponding Windows Event Logs dashboard
  • dnp3_control.log now includes clear_bit field to indicate if control code clear bit is set or unset
  • improved shared-object-creation.sh's cURL commands so that import failures for OpenSearch/Elasticsearch shared objects are printed to the debug logs rather than being redirected to /dev/null
• ✅ Component version updates
  • Arkime to v5.6.1
  • capa to v9.0.0
  • Fluent Bit to v3.2.7
  • OpenSearch and OpenSearch Dashboards to v2.19.0
    • includes workaround for [BUG] HTTP REST API hangs with Accept-Encoding zstd (opensearch-project/OpenSearch#17339) by disabling http.compression for internal connections to the OpenSearch API
• 🐛 Bug fixes
  • ANSI color codes from croc displayed in ssl-client-transmit (#559)
  • clear screen after auth_setup when using Dialog mode (#574)
  • warn and prompt user before changing NetBox database passwords out from underneath existing database (#565)
  • UFW software firewall for Malcolm ISO should automatically open ports for syslog (#560)
    • removed default port allowances (e.g., 5044/tcp, 9200/tcp, etc.) so that they could be set dynamically as part of configuration
• 📄 Configuration changes (in environment variables in ./config/) for Malcolm and in control_vars.conf for Hedgehog Linux
  • added functionality to control script to migrate environment variables between malcolm versions (e.g., moving environment variables from one .env file to another, removing deprecated/unused environment variables from .env files, etc.) by specifying these actions in config/env-var-actions.yml
  • removed CONNECTION_SECONDS_SEVERITY_THRESHOLD from lookup-common.env in favor of the new variables in zeek.env outlined below
  • increased defaults for NETBOX_CACHE_SIZE (to 10000) and NETBOX_CACHE_TTL (to 300 seconds) in netbox-common.env in relation to #547
  • some standardization of NetBox-related environment variables to reduce duplication across different .env files
    • DB_HOST and DB_NAME were moved from netbox.env to netbox-postgres.env
    • DB_PASSWORD and DB_USER were moved from netbox-secret.env to netbox-postgres.env
    • REDIS_CACHE_HOST and REDIS_HOST were moved from netbox.env to redis.env in relation to #580
    • REDIS_CACHE_PASSWORD and REDIS_PASSWORD were moved from netbox-secret.env to redis.env
    • removed netbox-redis-cache.env in relation to #580
    • renamed netbox-redis.env to redis.env in relation to #580
    • added MALCOLM_NETWORK_INDEX_ALIAS and MALCOLM_OTHER_INDEX_ALIAS to opensearch.env
      • these variables define the names for index aliases used in the index templates that are necessary for OpenSearch security analitics detectors
    • added the following variables to zeek.env for configuring the Zeek long connections plugin
      • ZEEK_LONG_CONN_REPEAT_LAST_DURATION (default true)
      • ZEEK_LONG_CONN_DO_NOTICE (default true)
      • ZEEK_LONG_CONN_DURATIONS (default 600,1800,3600,43200,86400)
• 🧹 Code and project maintenance
  • use arm-hosted runners for GitHub build actions for arm64 images (#557)
  • decouple redis from netbox (#580)
  • document standards for supply chain and code provenance checking (#555)
  • document incorporating new Suricata rules (and removing old ones) without restarting the Suricata containers (#589)
  • updates to documentation for Docker-based installation examples (#506)
  • some minor updates to .gitignore and .dockerignore files
  • standardize Dockerfiles to use ADD syntax instead of COPY
  • some standardization for the location scripts to which scripts are installed in Docker images
  • other minor documentation fixes

Malcolm is a powerful, easily deployable network 🖧 traffic analysis tool suite for network security monitoring 🕵🏻‍♀️.

Malcolm operates as a cluster of containers 📦, isolated sandboxes which each serve a dedicated function of the system. This makes Malcolm deployable with frameworks like Docker 🐋, Podman 🦭, and Kubernetes ⎈. Check out the Quick Start guide for examples on how to get up and running.

Alternatively, dedicated official ISO installer images 💿 for Malcolm and Hedgehog Linux 🦔 can be downloaded from Malcolm's releases page on GitHub. Due to limits on individual files in GitHub releases, these ISO files have been split 🪓 into 2GB chunks and can be reassembled with scripts provided for both Bash 🐧 (release_cleaver.sh) and PowerShell 🪟 (release_cleaver.ps1). See Downloading Malcolm - Installer ISOs for instructions.

As always, join us on the Malcolm discussions board 💬 to engage with the community, or pop some corn 🍿 and watch a video 📼.

[riley611]

what's the smoothest way to upgrade from 25.01 to current?

[mmguero]

TBH the upgrade process is kind of a pain, but see this link. This is something we want to improve hopefully this year.