Proposal: Monitoring which indexes are in readonly

[juanmancebo]

Abstract

When you want to monitor all readonly indexes you can't see which index(es) is/are in readonly and you have to search it manually.
i.e.

# ./check_es_system.sh -H ${host} -P ${port} -S -u ${user} -p ${pass} -t readonly
ES SYSTEM CRITICAL -  1 index(es) found read-only -

Proposal

If you parse the json you can extract the index(es) that is/are in readonly.
For example with this code snippet you could achieve that with jq. I tried it with the json_parse wrapper but it doesn't work. I guess the wrapper have to be adapted to work.

      if [[ $rocount -gt 0 ]]; then
        if [[ "$index" = "_all" ]]; then
          roindexes=$(jq -r '.[].settings.index |select(.blocks.read_only == "true").provided_name' <<< ${settings})
          output[${icount}]=" $rocount index(es) found read-only. ${roindexes} -"

[Napsty]

That's correct, the index name is not showing. That's the reason why I adjusted the output on the readonly check on _all indices.

It would be nice when the plugin shows which indices are set to read only, agree.

[Napsty]

Hi @juanmancebo

Thanks for the suggested code!

I built it into the branch issue-30. Can you try it please? https://github.com/Napsty/check_es_system/tree/issue-30

Example here:

$ /usr/lib/nagios/plugins/check_es_system.sh -H localhost -t readonly -u elastic -p secret -X jq
ES SYSTEM CRITICAL -  1 index(es) found read-only claudiotest - 116 index(es) found read-only (allow delete) logstash-2020.09.21 logstash-2020.09.04 rancher-et-2020.09.11 rancher-2020.09.12 rancher-vamp-2020.09.21 rancher-et-2020.09.19 rancher-mh-2020.09.04 rancher-mh-2020.09.20 filebeat-2020.09.04 rancher-mh-2020.09.10 rancher-2020.09.18 rancher-vamp-2020.09.04 rancher-et-2020.09.13 rancher-mh-2020.09.18 rancher-vamp-2020.09.08 .monitoring-es-6-2020.09.21 rancher-vamp-2020.09.13 rancher-et-2020.09.16 rancher-vamp-2020.09.06 idx filebeat-2020.09.08 rancher-mh-2020.09.15 rancher-et-2020.09.20 filebeat-2020.09.07 rancher-et-2020.09.08 filebeat-2020.09.01 claudiotest2 rancher-et-2020.09.12 rancher-vamp-2020.09.14 .management-beats rancher-vamp-2020.09.09 rancher-et-2020.09.17 rancher-mh-2020.09.14 rancher-et-2020.09.21 rancher-vamp-2020.09.10 logstash-2020.09.05 rancher-2020.09.09 rancher-2020.09.11 rancher-2020.09.20 rancher-2020.09.03 filebeat-2020.09.10 rancher-2020.09.14 rancher-vamp-2020.09.19 logstash-2020.09.06 rancher-mh-2020.09.06 rancher-vamp-2020.09.18 rancher-vamp-2020.09.07 rancher-et-2020.09.10 .triggered_watches-6 rancher-mh-2020.09.09 rancher-vamp-2020.09.11 logstash-2020.09.02 rancher-et-2020.09.09 filebeat-2020.09.21 logstash-2020.09.03 rancher-vamp-2020.09.01 rancher-mh-2020.09.02 rancher-mh-2020.09.13 rancher-mh-2020.09.12 rancher-2020.09.05 rancher-vamp-2020.09.15 rancher-et-2020.09.04 rancher-2020.09.02 rancher-2020.09.10 rancher-mh-2020.09.19 rancher-vamp-2020.09.02 .monitoring-kibana-6-2020.09.21 rancher-2020.09.16 rancher-mh-2020.09.05 rancher-vamp-2020.09.20 rancher-vamp-2020.09.03 filebeat-2020.09.03 rancher-et-2020.09.01 rancher-2020.09.15 rancher-2020.09.17 rancher-2020.09.19 rancher-mh-2020.09.03 logstash-2020.09.01 rancher-2020.09.08 rancher-vamp-2020.09.12 logstash-2020.09.08 rancher-mh-2020.09.11 rancher-et-2020.09.06 rancher-et-2020.09.03 kibana_sample_data_logs rancher-mh-2020.09.16 rancher-2020.09.07 rancher-et-2020.09.14 .kibana_7 rancher-vamp-2020.09.17 .kibana_task_manager rancher-mh-2020.09.07 filebeat-2020.09.05 rancher-2020.09.21 .security-6 .watches rancher-2020.09.06 filebeat-2020.09.11 filebeat-2020.09.09 rancher-2020.09.13 rancher-vamp-2020.09.05 rancher-et-2020.09.07 rancher-mh-2020.09.08 rancher-et-2020.09.18 rancher-mh-2020.09.21 rancher-et-2020.09.15 .kibana-6 filebeat-2020.09.02 rancher-2020.09.01 logstash-2020.09.07 rancher-mh-2020.09.17 rancher-et-2020.09.02 filebeat-2020.09.06 rancher-et-2020.09.05 rancher-vamp-2020.09.16 rancher-2020.09.04 

It only works when the parser is set to jq. I'd like to find a way how to do this in jshon, too, but was unable to find such a function.

A possible suggestion for jshon was made in keenerd/jshon#36. But this issue is already years old. Even if such a "search" parameter would be added in upstream, the compatibility with packages installed in the end user systems is not given.

[Napsty]

@juanmancebo Were you able to test?

[juanmancebo]

Hi @Napsty

I don't have an ES cluster right now to test this :(
I think it would be better to release a new major version removing the jshon support. I think maintain both JSON parsers will be very difficult and this lies in problems like this.