Crash on no Date: migrated from logstash#1806

[suyograo]

From @plarivee

When a mail does not have a Date header, the plugin crash.

{:timestamp=>"2014-09-29T16:43:12.059000-0400", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n  Plugin: <LogStash::Inputs::IMAP host=>\"mx.xxxxx.com\", tags=>[\"imap_mail\"], user=>\"[email protected]\", content_type=>\"text/plain\">\n  Error: undefined method `to_time' for nil:NilClass", :level=>:error}

seems to be link to line 108-109 of lib/logstash/inputs/imap.rb

  # Use the 'Date' field as the timestamp
  event["@timestamp"] = mail.date.to_time.gmtime

I think there should be
1- try / catch
2- Maybe try the Received header if no Date. I think the Imap class has something extracting date already. ( Tried to play with it, but couldn't get it to work as I don't know ruby much )

https://github.com/mikel/mail/blob/08cd8f5d9cd89ac2d23eab09aa57f96af735c234/lib/mail/elements/received_element.rb

Here is a test mail failing.
https://gist.githubusercontent.com/anonymous/4de2e0490ca32150cba8/raw/33aa686a8a1bfd12b84185425a61cf5f7496e485/gistfile1.txt

[suyograo]

From @plarivee

Maybe have a catch try for the encoding?

:timestamp=>"2014-09-30T15:08:52.236000-0400", :message=>"A plugin had an unrecoverable error. Will restart this plugin.\n Plugin: <LogStash::Inputs::IMAP host=>"mx.xxxxx.com", tags=>["xxxxxx"], user=>"[email protected]", content_type=>"text/plain">\n Error: undefined method `encode' for nil:NilClass", :level=>:error}

[radosevic]

anyone resolved this on version 1.4.2 ?
@suyograo did you manage to input emails?
thanks

[jordansissel]

I've been working on the imap input this past week and will see about adding this improvement.

What should Logstash use instead of the Date field for the timestamp if it doesn't exist?

[plarivee]

I mentioned in my initial case :

2- Maybe try the Received header if no Date. I think the Imap class has something extracting date already. ( Tried to play with it, but couldn't get it to work as I don't know ruby much )

[plarivee]

As a temp fix, I added

if mail.date.nil?
event.timestamp = LogStash::Timestamp.new()
else
event.timestamp = LogStash::Timestamp.new(mail.date.to_time)
end

From what I understand that puts "now" as the timestamp.

[untergeek]

@plarivee If you wouldn't mind submitting that as a pull request, it might be worth adding.

[waberc]

We are also facing this issue when the mailbox is loaded with many mails, still with version logstash-input-imap-3.0.6:

"A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::IMAP check_interval=>30, secure=>true, type=>\"imap\", delete=>false, fetch_count=>50, password=><password>, verify_cert=>false, folder=>\"INBOX\", content_type=>\"text/html\", port=>993, host=>\"mail-postfix.dev.local\", id=>\"1b8669fd7bc59b03d068f51e6904ff086da9c1404ecf21a880e95326deec36d2\", user=>\"[email protected]\", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>\"plain_4b96e0cd-b7ad-4f8f-b63e-915530ebab04\", enable_metric=>true, charset=>\"UTF-8\">, lowercase_headers=>true, expunge=>false, strip_attachments=>false>
  Error: undefined method `to_time' for nil:NilClass
  Exception: NoMethodError
  Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:124:in `block in parse_mail'
/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:47:in `block in decode'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-codec-plain-3.0.6/lib/logstash/codecs/plain.rb:35:in `decode'
/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:45:in `block in decode'
org/logstash/instrument/metrics/AbstractSimpleMetricExt.java:45:in `time'
org/logstash/instrument/metrics/AbstractNamespacedMetricExt.java:44:in `time'
/usr/share/logstash/logstash-core/lib/logstash/codecs/delegator.rb:44:in `decode'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:122:in `parse_mail'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:88:in `block in check_mail'
org/jruby/RubyArray.java:1792:in `each'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:82:in `block in check_mail'
org/jruby/RubyArray.java:1836:in `each_slice'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:80:in `check_mail'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:69:in `block in run'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:20:in `interval'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-imap-3.0.6/lib/logstash/inputs/imap.rb:68:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'"

[plarivee]

@waberc seems that no fix was ever implemented. You can either use the patch I did to prevent crashing. I no longer work with logstash-imap plugin so cannot spend time doing a proper try catch. But this patch just puts the current time as the date if no date.