React appropriately to HTTP 429 "rate limiting" response code

[nhooey]

I'm getting an HTTP 429 from HipChat, which the documentation says is rate-limiting. It seems the Sensu plugin doesn't know about this response code:

{
  "timestamp": "2016-04-01T17:20:25.679807+0000",
  "level": "info",
  "message": "handler output",
  "handler": {
    "command": "handler-hipchat.rb",
    "type": "pipe",
    "name": "hipchat"
  },
  "output": [
    "/opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/hipchat-1.5.1/lib/hipchat/room.rb:142:in `send': Unexpected 429 for room `2360342' (HipChat::UnknownResponseCode)
     from /opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/sensu-plugins-hipchat-0.0.3/bin/handler-hipchat.rb:70:in `block in handle'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:88:in `block in timeout'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:32:in `block in catch'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:32:in `catch'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:32:in `catch'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:103:in `timeout'
     from /opt/sensu/embedded/lib/ruby/2.2.0/timeout.rb:124:in `timeout'
     from /opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/sensu-plugins-hipchat-0.0.3/bin/handler-hipchat.rb:68:in `handle'
     from /opt/sensu/embedded/lib/ruby/gems/2.2.0/gems/sensu-plugin-1.2.0/lib/sensu-handler.rb:55:in `block in <class:Handler>'"
  ]
}

[majormoses]

Basically we need to look at the response code and if its a 429 then we need a exponential backoff and retry strategy.

[majormoses]

I just realized that looking at the stack trace that is something that they should probably be handled by the upstream library. I need to do some research but this is actually a bug on their end from what I can tell so far.