v4.8 - It works!

Release Notes

This version of the Cacti NetFlow plugin is a major milestone for this plugin. With this release, we now have full compatibility for both the NetFlow v9 and IPFIX protocols. We are currently not storing all template data from these protocols, but we are looking for ideas from the community. However, with that said, you will find the stream receivers called Listeners in the plugin, quite stable. If you are already running this plugin, take note that the service definitions change. So, you will have to pretty much re-install the service.

The amount of time that I (@TheWitness) and several other testers and Cacti Group QA personnel have put into this release is pretty enormous. Well over 4 man months of effort have been put in on nights and weekends, with people testing and providing feedback from around the world. Their contributions have increased first past quality on this plugin quite a bit.

There are some major features in the release that I would like to call out separate from the ChangeLog below. They include:

• Parallel Query - If you have NVMe and a lot of cores, or a scale out MaxScale install, you will be able to cut through several GB/second of NetFlow data per second with this version.
• Support for MariaDB MaxScale - If you have lot's of servers, this thing is faster than Elastic or Splunk. No kidding.
• Support of the MariaDB Aria Storage Engine - The Aria Storage Engine was designed for Analytics workloads an specifically tuned for highly performing Group By queries. It's a beast. If you are using MySQL 8++, you can use MyISAM for similar performance.
• Map Reduce Query Engine - So, you get native Parallel Query and Map Reduce to make it even faster.
• Shard Caching - With a pre-defined filter, you only have to search a partition once, and then never have to touch the raw data again. So, this version is super fast.
• Proper filtering for both IPv4 and IPv6 IP addresses.
• New Tree Map Chart Type
• Incorporation of Internet Registry Ingestion from all the Major Internet Router Registries
• Reliable Connections to ARIN through Web Services and RADB or others via WhoIs to gather information about flows.

The list of features goes on an on with this release. I hope you enjoy the release.

Personal Notes and Dedication

I decided years ago, that if I was not going to be a millionaire with Cacti, I might as well continue to pursue it as a philanthropic endeavor on my part, and this plugin shows that commitment to the FOSS community. I'm also contributing this plugin to the FOSS community and dedicating it to my baby sister, who is presently terminally ill with Stage IV cancer. I'm hoping for a miracle for her and I'm optimistic, but with that said, this plugin development gave me the opportunity to not loose my mind thinking about her predicament. You know, no one gets out of this world alive, but cancer is no way to go. I pray for her each and every day.

Anyway, off to the business of getting this thing released. You can see the by Feature and Issue list below, this has been a huge effort. It's really my biggest Free-and-Open work since MacTrack that I wrote in originally in 2004. Of course there is IBM's RTM, which is massive in it's own right, but this guy is FOSS from stem to stern.

So, enjoy, and I hope you enjoy using Cacti.

Regards,

Larry Adams
TheWitness

Changelog

--- 4.8 ---

• issue#66: Cacti flowview collector errors
• issue#71: Flowview no data were displayed
• issue#73: Timespan never saved when editing a filter
• issue#75: Cacti PHP Errors when select report filter Source AS
• issue#80: Error messages when generating a report with the flowview plugin
• issue#83: Cacti 1.2.23 and Flowview dev 3.3
• issue#84: Support more than one router per FlowView listener
• issue#85: Report Filter Listener Print All Rows Not Working
• issue#88: Create new Custom Flow Viewer Report
• issue#95: Multiple Deprecation errors associated with PHP8.x
• issue#100: Change the name of config.php to something like config.php.template
• issue#106: Properly detect the Cacti base on Ubuntu and Debian
• issue#118: Protocol filter not working
• issue#121: When you return to the FlowView tab, your navigation will be broken till the first click
• issue#132: The CIDR syntax filtering does not return correct results
• issue#139: Remove the source port from the ex_addr as it breaks multiple things
• issue#141: Using specific DNS servers results in unpack errors.
• issue: Flowview DNS Setting removed by accident
• issue: When changing the partition type, it does not take from the GUI right away
• issue: Sort field was being lost between selections
• issue: Dont fail to find flowview_connect() if the plugin is disabled
• issue: Make Domains only Domains/Hostnames only to remove confusion
• issue: Rework the SystemD startup process to use Cacti registration and unregistration functions
• issue: Allowed incoming address range was not working
• feature: Make it optional to query Arin to track Big Tech
• feature: Allow FlowView to remember your last filter when re-entering the FlowView page
• feature: Allow selection of the listener on the FlowView filter
• feature: Redesign flow_collectors.php for formalize support for IPFIX in FlowView
• feature: Store peers that connect to listeners for review and tagging
• feature: Inform the user of how many listener streams are present and their status
• feature: Track detected template definitions in the database and provide a way to view them
• feature: Add color to Bar Charts
• feature: Disable Charts when viewing printed reports
• feature: Show visual indicator that the filter has been saved
• feature: Add SIGHUP signal handling to flow-capture and flow_collector.php
• feature: Add support for Net/DNS2 to improve alternate DNS resolution support
• feature: Allow DNS Cache to be purged with a single button press.
• feature: Append local domain to locally resolved addresses from the system DNS
• feature: Show supported or not supported status when viewing Flow Templates
• feature: Import all Internet Route Registry databases when they change
• feature: Support Notification Lists for Email
• feature: Support a Report Generation Timeout
• feature: Support running Email Reports in Parallel
• feature#9: Support Pie Charts Instead of Bar
• feature#97: Add FreeBSD service control
• feature#111: Add a Template Export Button to make sharing Template information with the Cacti Group Easy
• feature#112: Provide a means to view the DNS Cache and Prune/Edit Entries
• feature#113: Provide buttons 'New', 'Rename', 'Delete' from the FlowView Tab
• feature#114: Provide an External Address dropdown in the Filter Definition
• feature#116: Have flow-capture automatically restart collectors that have crashed
• feature#117: Allow guest account access
• feature#120: Save Charting Options when you Save a Filter
• feature#122: Charts should have a Graph Title that Matches the Report Type
• feature#123: Support Setting the Default Chart Height
• feature#124: Support Treemap Charts instead of Bar
• feature#125: Perform Schema Upgrades in the Background
• feature#126: Cache ARIN Responses for reference
• feature#127: Show stream FlowView versions being ingested
• feature#128: Flowview attempts to import all streams even those that are reporting other information that is not supported by flowview
• feature#143: Support tcp listener
• feature#144: Allow mapping of a local subnet to a named private domain for home users
• feature#145: Add the V9/IPFIX Template IDs to the flowview raw tables
• feature#146: Use prepared statements for Flowview Filters to protect against SQL injections
• feature#149: Expose the ex_addr as a filter option and resolve it through DNS
• feature#150: Introduce Parallel Query - Map Reduce to FlowView
• feature#154: Support Parallel Queries through a MariaDB or MySQL proxy such as MaxScale
• feature#155: Allow Choosing your Raw Storage Engine Format
• feature#156: Create Persistent Shard Result Cache
• feature#160: Use whois providers such as radb.net to find origin AS when Arin does not provide it
• feature#163: Restructure the DNS Cache page to include: ASNs, People, Routes, etc. per the Internet Routing Registry format RSP
• feature#164: Only update the Routes databases if they have changed as recommened at radb.net
• feature#179: Provide Link in Cactis Graph View to Flowview
• feature#180: Allow Report Raw Data to be Downloaded
• feature#181: Allow Raw Report Data to be Exported from FlowView Page
• feature#191: Make it possible to disable a Listener Service
• feature#192: Allow an Administrator to observe the backlog on their Flowview Ports to know of overload

v3.3

--- 3.3 ---

• issue#61: IPFIX Errors in Cacti Log
• issue#62: Problem with ` symbol in SQL queries
• issue: Upgrade was running repeatedly due upgrade using legacy variable
• feature: Update FlowView to use Bulletin Board Charts
• feature: Minimum Cacti requirement 1.2.17 for Bulletin Board

v3.2

Release to fix issues with unique keys that could result in a loss of flow data.

--- 3.2 ---

• issue: Flowview raw table has invalid unique key that blocks the insert of data

v3.1

Details

Fixing mostly query issues in this release as well as a bug with enabling the plugin due to the inclusion of Cacti format file support. This release will be it for a while unless someone can find and "fix" the issues, of which I know there are some. Could use some help on the DNS side. For IP addresses that don't register their domains, the plugin dynamically goes to ARIN @ https://www.arin.net/ and parses the registered company name and "fakes" a domain. Any help on that and better DNS caching would be awesome!

Changelog

--- 3.1 ---

• issue#58: Flowview cannot enable in Cacti
• issue#59: Flowview Data wont display in CACTI
• issue: Results Cache not working
• feature: Support Cacti Format Files in FlowView

v3.0

This is a pretty major release of FlowView. All reported issues with the V9 and IPFIX FlowView versions have been resolved. In addition, the interface should be fully functional now as well as backend reporting. ChangeLog below. If you receive errors about an invalid mix of collations or charsets, run the repair_tables.php script after upgrading.

--- 3.0 ---

• issue#31: In systemctl status : CMDPHP: ERROR: A DB Exec Failed!, Error: Unknown column 'INF' in 'field list'
• issue#32: Netflow v9 - Netflow source not providing either prefix or nexthop information
• issue#33: cannot view flows
• issue#34: recurring updates for plugin_flowview_ports
• issue#35: Error when flowview raw tables do not exist
• issue#36: function flowview_get_owner_from_arin should be split for ipv4/ipv6
• issue#39: Using v9 Netflow source, flowview reports errors
• issue#40: Creating Flowview filters fails
• issue#41: Enhancement - Unnecessary Dialog Box: "Opeartion Successful - Select a filter to display data."
• issue#42: Firewall opened, Listener reports Down, tcpdump showing data, Systemd service running
• issue#43: NaN and Division by Zero errors
• issue#44: Error after install 'flowview': sizeof()
• issue#45: FlowView tab missing
• issue#46: Schedules dont work
• issue#47: flowData not showing data
• issue#48: Missing Code in ip_filter function
• issue#49: Flowview IPFIX throwing errors
• issue#50: Flowview errors cause it to become automatically disabled
• issue#51: FlowView Throws Errors When No Filters Exist
• issue#52: init.d script kills poller
• issue#53: Division by zero errors viewing tables
• issue#55: Netflow V9 Errors on CACTI
• issue#56: Sort Fields for Filters Not Saved Correctly
• issue: Partition tables were not being pruned.
• issue: Make the flow_collector.php resilient to loss of the database server.
• feature: New database design. Support for v5, v9, IPFIX flows transparently
• feature: Remove Open Flash Charts and use C3 Charts instead
• feature: Reworked user interface
• feature: Units files for systemd systems for flow-capture service
• feature: Remove use of Flow Tools
• feature: Support IPv4 and IPv6
• feature: Support daily and hourly partitioning
• feature: Run Schedules in Background
• feature: Allow Saving of some Filter Information from Flowview

v2.2 - Giving Open Flash Charts a Little more Leg

This is a fairly major update to the Flowcharts plugin. The database is now being used to hold session data, which should make things a bit faster. In the next release, we will remove Open Flash Charts, but in the mean time, it should be pretty compatible all the way up to the PHP7.x series and with the Firefox browser that has built in flash emulators.

I hope to move to a JavaScript/Canvas based graph engine shortly.

--- 2.2 ---

• issue#27: Deprecated functions and legacy path causing issues with schedules
• issue#28: Flowview filters not searchable using some international characters
• feature: Major rework of logic and code, make OpenFlashCharts work a little longer

v2.1

--- 2.1 ---

• issue: Prepare for sunrise theme in 1.1.17
• issue: Clean up the filter logic to preserve values
• issue: Make the graph size auto-detect screen size
• issue: Make reports sort properly