IPFIX multiple identical fields (Was: Can't decode flowset id 258 from observation domain id 256)

[AshHaque]

For IPFIX exporter (Cisco router of 4321 model and IOS 16), I am getting this message. I run the flow for hours. But this message is not going away. Using elastiflow on top this codec.

Netflow version 9 is working fine. Problem is only with IPFIX.

logstash version : 6.4
logstash-codec-netflow: 4.2

I am new in ELK. Help will be appreciated. I attached a PCAP file if it helps.

colopcap.zip

[AshHaque]

When this pcap was taken I was getting error message with flowset id 257.

[AshHaque]

here's the latest pcap from logstash.

colo_3010.zip

[AshHaque]

This is the debug log:

[2018-10-30T16:15:43,884][ERROR][logstash.inputs.udp ] Exception in inputworker {"exception"=>#<NameError: field 'ciscoAppHTTPHost' in BinData::Struct, is defined multiple times.>, "backtrace"=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:409:in block in ensure_field_names_are_valid'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:399:in ensure_field_names_are_valid'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:375:in block in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:266:in block in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:283:in sanitize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:264:in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:369:in sanitize_fields'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/struct.rb:345:in sanitize_parameters!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:302:in sanitize!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:210:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/sanitize.rb:192:in sanitize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/base.rb:302:in extract_args'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/base.rb:249:in extract_args'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/base.rb:81:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/warnings.rb:21:in initialize_with_warning'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:603:in do_register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:569:in block in register'", "org/jruby/ext/thread/Mutex.java:148:in synchronize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:568:in register'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:306:in block in decode_ipfix'", "org/jruby/RubyKernel.java:1114:in catch'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:290:in block in decode_ipfix'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:in block in each'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:289:in decode_ipfix'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:105:in block in decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:in block in each'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.4/lib/bindata/array.rb:208:in each'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-4.1.2/lib/logstash/codecs/netflow.rb:104:in decode'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:151:in inputworker'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:63:in block in run'"]}

[AshHaque]

In a single flowset Logstash is getting type 12235 (ciscoAppHTTPHost) multiple times. I thing this is the problem.

How to fix this?

[jorritfolmer]

There is no easy fix. The library we use to parse doesn't support multiple identical fields.
Similar issues for reference: #93 #142

[AshHaque]

Thanks for the update. Apart from this issue my setup is running fantastic. Waiting for the fix to play with IPFIX. Just asking if there is any work in progress on it?

[jorritfolmer]

No progress, sorry.

[dmittendorf]

@jorritfolmer I ran into this same issue when trying to use OpenVSwitch as an IPFIX source, since it duplicates the interfaceName fields.

I have a working patch that addresses this problem by pre-processing the fields in the template received from the source and "hides" the duplicate/identical fields by replacing the field name with an empty string before constructing the BinData::Struct from the template fields. This allows templates with duplicate fields to be successfully processed/loaded, however, the side affect is that duplicate values received from the source will be ignored and won't be passed through in the generated events.

This seems like a reasonable trade-off, and the code change to support this is very small.

If you think this is a reasonable approach, I'll go ahead and create supporting tests and a PR for this change.

[jorritfolmer]

Yes that sounds like an improvement over the current state.
It doesn't get us towards IPFIX RFC compliance, see #83, because there it states in chapter 8:

Collecting Processes MUST properly handle Templates with multiple identical Information Elements.

I'm no longer maintaining logstash-codec-netflow through, but I would suggest you create a PR and go from there.

[ramrode]

am facing the same issue as @dmittendorf and looking for a solution .

@dmittendorf can you please share your solution ?