Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't (yet) decode flowset id 274 from source id 256 - Cisco ASR 1001-X #186

Open
imuab opened this issue Oct 8, 2019 · 7 comments
Open

Can't (yet) decode flowset id 274 from source id 256 - Cisco ASR 1001-X #186

imuab opened this issue Oct 8, 2019 · 7 comments

Comments

@imuab
Copy link

imuab commented Oct 8, 2019 •
edited
Loading

Hello,

I have some issues with logstash Netflow codec and Cisco ASR 1000.
I am using Netflow Version 9 and have following messages in my logstash logs:

[2019-10-08T15:36:22,517][ERROR][logstash.inputs.udp ] Exception in inputworker {"exception"=>java.lang.ClassCastException: class org.jruby.gen.RubyObject4 cannot be cast to class org.jruby.RubyFixnum (org.jruby.gen.RubyObject4 is in unnamed module of loader org.jruby.util.OneShotClassLoader @6b6def36; org.jruby.RubyFixnum is in unnamed module of loader 'app'), "backtrace"=>["org.jruby.runtime.invokedynamic.MathLinker.fixnum_op_equal(MathLinker.java:237)", "java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)", "org.jruby.runtime.invokedynamic.MathLinker.fixnumOperator(MathLinker.java:171)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:171)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.exceptions.CatchThrow.enter(CatchThrow.java:32)", "org.jruby.RubyKernel.rbCatch19Common(RubyKernel.java:1197)", "org.jruby.RubyKernel.rbCatch19(RubyKernel.java:1193)", "org.jruby.RubyKernel$INVOKER$s$rbCatch19.call(RubyKernel$INVOKER$s$rbCatch19.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroOrOneBlock.call(JavaMethod.java:577)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode_netflow9$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:167)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:177)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode_netflow9$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:166)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$block$decode$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:97)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:477)", "org.jruby.ir.targets.YieldSite.yield(YieldSite.java:105)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$block$each$1(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146)", "org.jruby.runtime.BlockBody.yield(BlockBody.java:114)", "org.jruby.runtime.Block.yield(Block.java:170)", "org.jruby.RubyArray.each(RubyArray.java:1800)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb:208)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bindata_minus_2_dot_4_dot_4.lib.bindata.array.RUBY$method$each$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bindata-2.4.4/lib/bindata/array.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:223)", "org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:230)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_codec_minus_netflow_minus_4_dot_2_dot_1.lib.logstash.codecs.netflow.RUBY$method$decode$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-netflow-4.2.1/lib/logstash/codecs/netflow.rb:93)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$inputworker$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:151)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$method$inputworker$0$VARARGS(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_input_minus_udp_minus_3_dot_3_dot_4.lib.logstash.inputs.udp.RUBY$block$run$2(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-udp-3.3.4/lib/logstash/inputs/udp.rb:63)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}

[2019-10-08T15:36:46,329][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 275 from source id 256, because no template to decode it with has been received. This message will usually go away after 1 minute.

  • Version: Logstash 7.4.0 / Netflow Codec 4.2.1
  • Operating System: Docker 1.13.1 on Centos 7
  • Config File (if you have sensitive info, please remove it):

input {
udp {
port => 2055
type => "netflow"
codec => netflow {
include_flowset_id => true
enable_metric => true
versions => [5, 9]
}
}
}

output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "netflow-%{+YYYY.MM.dd}"
user => elastic
password => changeme
}
stdout { codec => rubydebug }
}

  • Sample Data:
    grafik

grafik

  • Steps to Reproduce:
@novaksam
Copy link

novaksam commented Oct 8, 2019

Have the same issue since I upgraded to 7.4.0; I just switched back to 7.3.2 and it's working fine with the same Netflow-Codec version

@imuab
Copy link
Author

imuab commented Oct 9, 2019

@novaksam Did you just downgrade Logstash to 7.3.2 or all instances like elastic, kibana etc?

@imuab
Copy link
Author

imuab commented Oct 9, 2019 •
edited
Loading

I got it working with Logstash 7.3.2, thank you @novaksam!

But i got some problems with the built-in netflow visualizations and dashboards.
The visualizations are matched on an different index-pattern and i cant match them for my netflow-* index.

The dashboards are looking for data with the filter "input.type: netflow", but they cant find any data.
Are they any syntax problems in my netflow.conf? Im saying type => netflow , is that wrong?

When im looking into logs, the type is correct. It says netflow, as you can see.

grafik

I think I am missing some fields, right?

grafik

@smaxx1337
Copy link

Hey, we have the same problem here with decoding the template. We are also using a Cisco ASR 1001-X and Logstash 7.3.2.


[2019-10-10T09:03:47,120][WARN ][logstash.codecs.netflow  ] Can't (yet) decode flowset id 258 from source id 6, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-10-10T09:03:47,120][WARN ][logstash.codecs.netflow  ] Can't (yet) decode flowset id 258 from source id 6, because no template to decode it with has been received. This message will usually go away after 1 minute.
[2019-10-10T09:03:47,125][WARN ][logstash.codecs.netflow  ] Unsupported field in template 258 {:type=>44999, :length=>32}
[2019-10-10T09:03:47,125][WARN ][logstash.codecs.netflow  ] Can't (yet) decode flowset id 258 from source id 6, because no template to decode it with has been received. This message will usually go away after 1 minute.

pcap

@imuab
Copy link
Author

imuab commented Oct 10, 2019 •
edited
Loading

@smaxx1337 the logstash netflow module is deprecated in 7.4.0. It would recommand using the filebeat netflow module for the future. I just changed my setup as well and it's working absolutly fine with filebeat 7.4.0.

https://www.elastic.co/guide/en/beats/filebeat/7.4/filebeat-module-netflow.html

@robcowart
Copy link
Contributor

robcowart commented Nov 16, 2019 •
edited
Loading

@imuab Only the Netflow module (which was basically ElastiFlow 1.0.0) is deprecated, not the Logstash Netflow Codec.

@novaksam
Copy link

novaksam commented Dec 6, 2019

Looks like this will/should be fixed in LS 7.6
elastic/logstash#11196

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@novaksam @smaxx1337 @robcowart @imuab