Nagios not sending notifications due to missing ruby gem in nagios herald

[crohacz]

Hardware information
Linux nagios01 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Running on GCE.

Summary of the problem

Nagios is failing to send notifications with the simple use of a check_http command. Here is how I'm executing the command. The command executes fine and I see in Nagios that the state of my services switches from OK to Critical when the check_http command fails. All is good and well there.

What is happening

When a command on a services executes, it does not trigger notifications to its contacts or contact groups.

The logs show this:

/usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- chef/search/query (LoadError)
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/etsy/nagios-herald/lib/nagios-herald/helpers/ganglia_graph.rb:2:in `<top (required)>'
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/etsy/nagios-herald/lib/nagios-herald/helpers.rb:1:in `<top (required)>'
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/etsy/nagios-herald/lib/nagios-herald.rb:7:in `<top (required)>'
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib64/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/etsy/nagios-herald/bin/nagios-herald:3:in `<main>'

What I expect to happen
I expect an email notification specifically, but it doesn't seem to be triggering any type of notification.

Relevant Code

Apologies in advance for the bad formatting.

# check host for 200 OK, onredirecit is critical, ARG1=port, warn at 1s, critical at 8s, timeout at 8s
define command {
        command_name    check_random_host
        command_line    $USER1$/check_http -H $HOSTADDRESS$ -e "200 OK" -f critical -p $ARG1$ -w 1 -c 8 -t 8
}

#Host & hostgroups: 
define hostgroup {
    hostgroup_name      some-hostgroup
    alias                           hostgroup example
}

define host {
    use                          generic-host
    host_name              some-generic-host
    address                   DNS name to check
    notes                       A host definition used for the service
    check_command    dummy-hostup
    hostgroup               some-hostgroup
}

#Service: 
define service {
    use                                  local-service
    name                               fake-service
    service_description        Fake service description
    hostgroup                       some-hostgroup
    check_command           check_random_host!80
    check_interval               1
    retry_interval                  1
    max_check_attempts    4
    # one email per hour
    notification_interval      60
    notes                              URGENT
    notes_url                        some notes
    contact_groups             null
    contacts                        crohacz 
}

#contact: 
define contact { 
  contact_name                          crohacz
  alias                                          Christine Rohacz 
  email                                        ******@gmail.com
  use                                           generic-contact
}

How to reproduce
Triggering a notification will reproduce the error. Additionally, just running /usr/etsy/nagios-herald/bin/nagios-herald produces the same error.