Sensu checks are not executing on scheduled time

[Shivani3351]

Sensu checks are not executing occasionally on scheduled time

Expected Behavior

Checks should never stop being scheduled unless they are no longer published.

Current Behavior

Some sensu checks are not executing occasionally on scheduled time .

Sensu logs :
influxdb handler asset configured
{"component":"schedulerd","cron":"0 */2 * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-elkes-data-backup-validation","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"/35 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-netty-thread-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-basic-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-connection-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-mdm-ui-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-rdp-api-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-rdp-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-system-user-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 0 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"topology-apm-stats-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}

Possible Solution

Steps to Reproduce (for bugs)

Context

Some sensu checks are not executing occasionally on scheduled time .

influxdb handler asset configured
{"component":"schedulerd","cron":"0 */2 * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-elkes-data-backup-validation","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"/35 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-netty-thread-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:21Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-basic-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-connection-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-mdm-ui-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * ","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-rdp-api-server-status","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"/4 * * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tech-nginx-status-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:22Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-rdp-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 */12 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"tenant-system-user-validation-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}
{"component":"schedulerd","cron":"0 0 * * *","error":"error while starting ring watcher: context canceled","level":"error","msg":"error scheduling check","name":"topology-apm-stats-alert","namespace":"default","scheduler_type":"round-robin cron","time":"2024-02-07T16:27:23Z"}

Your Environment

• Sensu version used (sensuctl, sensu-backend, and/or sensu-agent):
  sensu-go : 6.10.0
  sensu-server :
  sensu-etcd : v3.5.0
• Installation method (packages, binaries, docker etc.):
  packages
• Operating System and version (e.g. Ubuntu 14.04):
  Debian

[elfranne]

I had a similar issue and I solved it by building a cluster. We have around 19.000 checks spread across 600 hosts. Made a cluster a 3 node backends and a 3 node etcd cluster. No issue since.