The Flow Analyzer is a Netflow, IPFIX, and sFlow collector and parser, available under the BSD 3-Clause License, that stores flows in Elasticsearch and visualizes them in Kibana. It is designed to run on Ubuntu Server, either as a single installation or as part of an Elasticsearch cluster.
Visualizations and Dashboards are provided to support network flow analysis right out of the box.
See the License section below for licensing details.
- Project Goals
- Features
- Requirements
- Installation
- Ports and Protocols
- Data Access
- Limitations
- Debugging
- Contributing
- License
- Attributions
Our goal is to provide superior Netflow and IPFIX collection, visualization, and analysis. We do that by creating:
- Efficient, accessible, and sustainable software
- Scalable solutions that can evolve as you grow
- Superior documentation from architecture through installation, configuration, tuning, and troubleshooting
One other goal is to make Elasticsearch and Kibana easy to implement and accessible to those who haven't used it before. The learning curve for distributed search systems and dashboarding software can be steap, but we think that everyone should be able to realize the benefits of meaningful, beautiful data visualization.
The Flow Analyzer has the flow collection, tagging, and categorizing capabilities to satisfy enterprise and service provider requirements.
Go from zero to visualized flow data in less than one hour. See the installation documentation for easy to follow step-by-step instructions.
The Manito Networks Flow Analyzer supports the following flow data protocols:
- Netflow v5 (Cisco)
- Netflow v9 (Cisco)
- IPFIX (IEEE, aka Netflow v10)
- sFlow (InMon Corporation)
- Traffic Flow (Mikrotik, Netflow-equivalent)
- Netstream (Huawei Technologies, Netflow-equivalent)
If you're not familiar with Netflow or IPFIX take a look at the Introduction to Netflow and IPFIX. For information on sFlow and supported sFlow structures see the Flow Analyzer sFlow document. Our software collects Netflow, IPFIX, and sFlow data and then parses and tags it. The parsed and tagged data is stored in Elasticsearch for you to query and graph in Kibana.
The Flow Analyzer supports all Netflow v5 fields, non-proprietary Netflow v9 fields, IPFIX fields specified in the RFC, and almost all sFlow structures defined by InMon Corporation's enterprise ID. See the Fields document for a description of Netflow (v5, v9) and IPFIX fields. The sFlow document includes descriptions of supported Flow and Counter structures.
Kibana Visualizations and Dashboards are included so you can leverage supported fields and structures right away.
Some limitations exist, mostly around proprietary or undocumented fields in Netflow and proprietary structures in sFlow. See the Limitations section for details. Efforts are made to skip over unsupported or proprietary elements and continue parsing data uninterrupted.
Our custom Netflow, IPFIX, and sFlow collectors ingest, parse, and tag flow data. We record not only the basic protocol and port numbers but we also take it a step further and correlate the following:
- IANA protocol numbers to names (eg protocol 1 to "ICMP", 6 to "TCP", 89 to "OSPF")
- IANA-registered port numbers to services (eg port 80 to "HTTP", 53 to "DNS")
- Services to categories (eg HTTP, HTTPS, Alt-HTTP to "Web")
This tagging functionality is running by default and happens transparently in the background. For more information on tagging functionality see the Tagging documentation.
A reverse lookup against observed IPs is done if DNS lookups are enabled. Resolved domains are cached for 30 minutes to reduce the impact on DNS servers. Popular domains like facebook.com and cnn.com are categorized with content tags like "Social Media" and "News" to provide insight into website browsing on the network.
Correlation of MAC address OUI's to top manufacturers is done to help graph traffic sources in hetergenous environments.
Note: This feature is in beta, and the list of OUI's to be built is quite extensive.
See the Requirements documentation for information on RAM, CPU, storage, and operating system requirements.
Clone the Git repo, run the installation script, and import the pre-built Kibana objects. See the installation documentation for more information.
All services listen for UDP flow packets on the following default ports:
| Service | Protocol | Default Port | Purpose |
|---|---|---|---|
| Netflow v5 | UDP | 2055 | Basic flow monitoring |
| Netflow v9 | UDP | 9995 | Intermediate flow monitoring |
| IPFIX | UDP | 4739 | Advanced flow monitoring |
| sFlow | UDP | 6343 | Advanced flow and performance monitoring |
Access your flow data however you want - Kibana dashboards, Elasticsearch JSON-formatted queries, or curl HTTP requests. Access to Kibana can be restricted using Squid via a reverse proxy and the directions for setting that up are included.
See the installation documentation for more information.
The following protocols, vendor features, or vendor-proprietary functions are NOT supported by the Flow Analyzer project.
These technologies may use a supported protocol for transport but there are proprietary fields, codes, or structures in use. Some protocols may require parsing that is undocumented or proprietary to the vendor.
- Cisco ASA Netflow Security Event Logging (NESL)
- Cisco NAT Event Logging (NEL)
- Cisco Streaming Telemetry
- Netflow v5 does not support IPv6.
If you run into any issues during or after installation check out the Debugging page for helpful commands and debugging options.
We encourage people who use the Flow Analyzer to contribute to the project if they find a bug or documentation issue, or want to see a feature added. See the Contributing page for more information about contributing code to the project. Interactions between participants should be within the bounds of the Code of Conduct.
Copyright (c) 2017, Manito Networks, LLC All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
-
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
-
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
-
Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Trademark, copyright, and license disclosures below:
- "Elasticsearch" and "Kibana" are registered trademarks of Elasticsearch BV.
- "Elasticsearch" and "Kibana" are distributed under the Apache 2 license by Elasticsearch BV.
- "Ubuntu" is a registered trademark of Canonical Ltd.
- "sFlow" is a registered trademark of InMon Corporation.
- "Cisco" is a registered trademark of Cisco Systems, Inc.
- "Mikrotik" is a trademark of Mikrotikls SIA.
- "Huawei" is a trademark of Huawei Technologies Co., Ltd.
- "NVIDIA" is a trademark of NVIDIA Corporation.
- "Broadcom" is a trademark of AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
Copyright (c) 2017, Manito Networks, LLC All rights reserved.