Added threshold for pods to be in the non ready state

Author: Daniel Andrzejewski

CHANGELOG.md

@@ -5,6 +5,11 @@ This CHANGELOG follows the format listed [here ](https://github.com/sensu-plugin
 
 ## [Unreleased]
 
+### Chaned
+ - `check-kube-pods-running.rb`: Skip a POD which is in the not ready state for shorter time than the specified time. Otherwise, the check alerts if we get lots of new PODs which are spawned every second and get up or get terminated longer than a minute. (@sys-ops)
+
+
+## [3.2.0] - 2018-11-21
 ### Changed
  - `check-kube-service-available.rb`: Skip a service if its selector is empty. Otherwise all PODs in the cluster are listed with client.get_pods() call (including those that we do not want to monitor) (@sys-ops)
 

bin/check-kube-pods-running.rb

@@ -34,6 +34,7 @@
 #                                      Exclude wins when a node is in both include and exclude lists
 #         --include-nodes              Include the specified nodes (comma separated list), an
 #                                      empty list includes all nodes
+#         --time TIME                  Threshold for pods to be in the non ready state
 #     -f, --filter FILTER              Selector filter for pods to be checked
 #     -p, --pods PODS                  List of pods to check
 # NOTES:
@@ -59,6 +60,12 @@ class AllPodsAreRunning < Sensu::Plugins::Kubernetes::CLI
          long: '--pods',
          default: 'all'
 
+  option :not_ready_time,
+         description: 'Threshold for pods to be in the non ready state',
+         long: '--time TIME',
+         proc: proc(&:to_i),
+         default: 300
+
   option :pod_filter,
          description: 'Selector filter for pods to be checked',
          short: '-f FILTER',
@@ -110,7 +117,10 @@ def run
       next if should_exclude_node(pod.spec.nodeName)
       next unless pods_list.include?(pod.metadata.name) || pods_list.include?('all')
       next unless pod.status.phase != 'Succeeded' && !pod.status.conditions.nil?
-      failed_pods << pod.metadata.name unless ready? pod
+      pod_stamp = Time.parse(pod.status.startTime)
+      if (Time.now.utc - pod_stamp.utc).to_i > config[:not_ready_time]
+        failed_pods << pod.metadata.name unless ready? pod
+      end
     end
 
     if failed_pods.empty?