Python problem: iupdate.src_ip/_SubnetTree.SubnetTree___getitem__(self, cidr)

[ogogon]

While Zeek is running, I periodically receive letters from him in the mail with the following content:
Subject: [Zeek] Connection summary from ... (then a time interval is specified)

Traceback (most recent call last):
  File "/usr/local/bin/trace-summary", line 1115, in <module>
    readConnSummaries(file)
  File "/usr/local/bin/trace-summary", line 508, in readConnSummaries
    parseConnLine(line, field_sep, unset_field, idx, max_idx_1, is_json, scope_separator)
  File "/usr/local/bin/trace-summary", line 844, in parseConnLine
    LocalNetsIntervals[iupdate.src_ip].update(iupdate)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/usr/local/lib/zeek/python/SubnetTree.py", line 103, in __getitem__
    return _SubnetTree.SubnetTree___getitem__(self, cidr)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd9 in position 0: invalid continuation byte
        0,06 real         0,06 user         0,00 sys

-- 
[Automatically generated.]

My Zeek is installed on a router under FreeBSD from ports (compiled from sources with automatic dependency control). I am ready to provide detailed information.

I created a topic on the forum, but there was silence there. https://community.zeek.org/t/whats-wrong-with-this-crawling-hose/7393

[awelzel]

@ogogon - which Python version are you using?

@bbannier - seems there's a good chance you fixed this with zeek/pysubnettree#38 (comment) ?

[ogogon]

@ogogon - which Python version are you using?

root@gw:/home/ogogon # uname -a
FreeBSD gw 13.2-RELEASE-p3 FreeBSD 13.2-RELEASE-p3 GENERIC amd64
root@gw:/home/ogogon # pkg info | grep py
py311-backports-1              Shared namespace shim for py-backports.* ports
py311-btest-1.1                Simple driver for basic unit tests
py311-build-1.2.1              PEP517 package builder
py311-configparser-3.5.3_1,1   INI style configuration file parser
py311-docutils-0.19,1          Python Documentation Utilities
py311-flit-core-3.9.0          Distribution-building parts of Flit
py311-gitdb-4.0.11_1           Git Object Database
py311-gitpython-3.1.30         Python Git Library
py311-installer-0.7.0          Library for installing Python wheels
py311-packaging-24.0           Core utilities for Python packages
py311-pyproject_hooks-1.1.0    Wrappers to call pyproject.toml-based build backend hooks
py311-semantic-version-2.10.0_1 Python library provides a few tools to handle SemVer in Python
py311-setuptools-63.1.0_1      Python packages installer
py311-smmap-5.0.1_1            Sliding-window memory map manager
py311-sqlite3-3.11.9_7         Standard Python binding to the SQLite3 library (Python 3.11)
py311-wheel-0.43.0             Built-package format for Python
py311-zkg-2.14.0               Zeek NSM package manager
python311-3.11.9               Interpreted object-oriented programming language
root@gw:/home/ogogon # python3.11 --version
Python 3.11.9
root@gw:/home/ogogon # python3.11 
Python 3.11.9 (main, Apr  9 2024, 03:27:27) [Clang 14.0.5 (https://github.com/llvm/llvm-project.git llvmorg-14.0.5-0-gc12386 on freebsd13
Type "help", "copyright", "credits" or "license" for more information.
>>> 
root@gw:/home/ogogon # 

[awelzel]

Thanks @ogogon - I'm rather confident the pysubnettree from Benjamin will fix the issue. I put it onto the list of backports for 6.0 and 6.2. I'm not sure how you're building, but would you be able to use the latest master version of zeekctl and see if that fixes it?

[ogogon]

I'm rather confident the pysubnettree from Benjamin will fix the issue. I put it onto the list of backports for 6.0 and 6.2.

Thank you. But I don’t really understand what it is and how to use it.

I'm not sure how you're building, but would you be able to use the latest master version of zeekctl and see if that fixes it?

I use FreeBSD Ports - this is a very convenient technology in which installation is done from source, with automatic dependency tracking.
Here is the installed version of your program and a list of dependencies.

ogogon@gw:/usr/ports/security/zeek/work/zeek-6.0.4# pkg info zeek
zeek-6.0.4
Name           : zeek
Version        : 6.0.4
Installed on   : Mon Jun 10 15:40:43 2024 MSK
Origin         : security/zeek
Architecture   : FreeBSD:13:amd64
Prefix         : /usr/local
Categories     : security
Licenses       : CC-BY-4.0
Maintainer     : [email protected]
WWW            : https://www.zeek.org/
Comment        : System for detecting network intruders in real-time
Options        :
	DEBUG          : off
	GEOIP2         : on
	IPSUMDUMP      : on
	LBL_CF         : on
	LBL_HF         : on
	MINSIZEREL     : off
	PERFTOOLS      : off
	RELEASE        : on
	RELWITHDEBINFO : off
	SPICY          : on
	ZEEKCTL        : on
	ZKG            : on
Shared Libs required:
	libpython3.11.so.1.0
	libmaxminddb.so.0
	libintl.so.8
	libcares.so.2
Shared Libs provided:
	libspicy.so
	libhilti.so
	libbinpac.so.0
Annotations    :
	FreeBSD_version: 1302001
	cpe            : cpe:2.3:a:zeek:zeek:6.0.4:::::freebsd13:x64
Flat size      : 150MiB
Description    :
Zeek (formerly known as Bro) is an open-source, Unix-based Network
Intrusion Detection System (NIDS) that passively monitors network
traffic and looks for suspicious activity. Zeek detects intrusions
by first parsing network traffic to extract its application-level
semantics and then executing event-oriented analyzers that compare
the activity with patterns deemed troublesome. Its analysis includes
detection of specific attacks (including those defined by signatures,
but also those defined in terms of events) and unusual activities
(e.g., certain hosts connecting to certain services, or patterns
of failed connection attempts).

Zeek is documented in the USENIX 1998 Security Conference proceedings
(as Bro).

ogogon@gw:/usr/ports/security/zeek/work/zeek-6.0.4# pkg info -dr zeek
zeek-6.0.4
Depends on     :
	lbl-hf-1.11
	lbl-cf-1.2.8
	bash-5.2.26_1
	py311-zkg-2.14.0
	libmaxminddb-1.10.0
	ipsumdump-1.86_2
	python311-3.11.9
	perl5-5.34.3_3
	c-ares-1.30.0
	gettext-runtime-0.22.5
	py311-sqlite3-3.11.9_7
ogogon@gw:/usr/ports/security/zeek/work/zeek-6.0.4# 

I would really hate to leave this paradigm and start installing something manually. Firstly, manually installed programs are not monitored for vulnerabilities. Secondly, they fall out of automatic version update mechanisms. Thirdly, you always need to remember which programs need to be added and from where, as dependencies; after some time, when reinstalling, this can become a problem.
It might be worth inviting the maintainer of your package in FreeBSD Ports to discuss the problem. This is Craig Leres [email protected].

[leres]

I guess I'm not seeing this because I have MailConnectionSummary=0 in zeekctl.cfg.

Is the change to SubnetTree_wrap.cc sufficient to fix this (for 6.0.4)? When I diff the version of pysubnettree that is bundled with zeek 6.0.4 with master/pysubnettree I see tons of unrelated changes.

[awelzel]

I see tons of unrelated changes.

Unfortunately, yes. Roughly the diff you see here and in SubnetTree.h:

https://github.com/zeek/pysubnettree/pull/38/files#diff-1ffeb27f5b366cf9a95e91ce7e4f076f77c94bacc9fdc7cbac220204742992b8