[BUG] .opensearch-sap-log-types-config and .opensearch-sap-pre-packaged-rules-config should not exist

[ryanqin01]

Describe the bug
Above two indices should not exist when starting cluster,

To Reproduce
Steps to reproduce the behavior:

1. Start cluster
2. cat shards

Expected behavior
If the cluster is just created, the indices should be clean.

[Mehdi-Bendriss]

In addition to the above, it seems that .plugins-ml-config was also added by default.
This prevent the cluster from being "green" on a single node cluster.

[ryanqin01]

In addition to the above, it seems that .plugins-ml-config was also added by default. This prevent the cluster from being "green" on a single node cluster.

found it also. I can understand the behavior, and I can tolerate index like .security. But too many default indices are bad experience

[anasalkouz]

Thanks @ryanqin01 for reporting. I have 2 follow up questions.
Could you please list the plugins that have been installed in the cluster?
Why do you think it's a bad experience to have those default indices?

[ryanqin01]

Thanks @ryanqin01 for reporting. I have 2 follow up questions. Could you please list the plugins that have been installed in the cluster? Why do you think it's a bad experience to have those default indices?

Hi Anas,

The plugins are just the default plugins if you download the opensearch 2.9.0 tar from official website.

In my opinion, the default thing should be simple and clean. There should not have too many plugins that most people may not need, and the indices should also be clean. Security index is necessary and understandable, but indices like -sap- is not. It even have company name here! We are an opensource project.
Another bad experience is, I use a single node cluster to test opensearch, but these plugins and their indices will cause the cluster status "yellow". If people just deployed a cluster, the cluster status is already yellow, it's obviously bad experience.

[Mehdi-Bendriss]

@anasalkouz is there a reason why the .opendistro_security has auto_expand_replicas: 0-all but not .plugins-ml-config which expects a > 1 number of nodes for this index to be green?

[TravAJohn]

I'm also experiencing this issue. I'm using a single node OpenSearch cluster with Graylog. At first I though it was something with how Graylog was handling the sharding but it seems other are experiencing this as well.

Here's the output:

curl -X GET "localhost:9200/_cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state&pretty"

.opensearch-sap-pre-packaged-rules-config 0     r      UNASSIGNED                  CLUSTER_RECOVERED
.opensearch-sap-log-types-config          0     r      UNASSIGNED                  CLUSTER_RECOVERED
.opensearch-sap-pre-packaged-rules-config 0     p      STARTED    gl_node 
gl-events_0                               0     p      STARTED    gl_node 
graylog_9                                 0     p      STARTED    gl_node 
graylog_8                                 0     p      STARTED    gl_node 
graylog_7                                 0     p      STARTED    gl_node 
.plugins-ml-config                        0     p      STARTED    gl_node 
.opensearch-observability                 0     p      STARTED    gl_node 
graylog_6                                 0     p      STARTED    gl_node 
.opensearch-sap-log-types-config          0     p      STARTED    gl_node 
gl-system-events_3                        0     p      STARTED    gl_node 
gl-system-events_6                        0     p      STARTED    gl_node 
graylog_10                                0     p      STARTED    gl_node 
gl-system-events_4                        0     p      STARTED    gl_node 
gl-system-events_5                        0     p      STARTED    gl_node 
graylog_11                                0     p      STARTED    gl_node

[anasalkouz]

Thanks @Mehdi-Bendriss and @TravAJohn for your inputs. I do agree the default opensearch distribution should has less number of plugins and all plugins should use auto_expand_replicas: 0-all

[anasalkouz]

@bbarani what do you think? who should drive this?

[BenoitPoulet]

Hello,

Same problem here, i just upgraded to 2.9.0, and now my single node OS is Yellow, and I don’t really known how to get it back to Green 😐

$ curl -X GET http://localhost:9200/_cat/shards?v  | grep UNASSIGNED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18168  100 18168    0     0  1774k      0 --:--:-- --:--:-- --:--:-- 1774k
.opensearch-sap-log-types-config          0     r      UNASSIGNED
.opensearch-sap-pre-packaged-rules-config 0     r      UNASSIGNED
.plugins-ml-config                        0     r      UNASSIGNED

Edit , in help found in this link opensearch-project/security#3130 (comment)
I removed the plugins :

/usr/share/opensearch/bin/opensearch-plugin remove opensearch-security-analytics
/usr/share/opensearch/bin/opensearch-plugin remove opensearch-ml

And I trashed the indices

curl -X DELETE http://localhost:9200/.opensearch-sap-log-types-config
curl -X DELETE http://localhost:9200/.opensearch-sap-pre-packaged-rules-config?pretty=true
curl -X DELETE http://localhost:9200/.plugins-ml-config?pretty=true

[dch]

I'm also a single-node graylog user (our logs are buffered to disk, so we don't need a complex HA setup), and updating opensearch introduced this issue.

thanks @BenoitPoulet for this work-around.

Can the config be changed in next release, to accommodate the single node config without needing this sort of workaround please?

[dblock]

Does it make sense to change the code that creates each of these indexes to only create them the first time actual data needs to be written into them?

[ryanqin01]

Does it make sense to change the code that creates each of these indexes to only create them the first time actual data needs to be written into them?

absolutely

[TravAJohn]

@anasalkouz @bbarani any update on the status of this issue?

[dblock]

@ryanqin01 AFAIK nobody is working on this, do you want to pick it up?

[ryanqin01]

@ryanqin01 AFAIK nobody is working on this, do you want to pick it up?

yeah sure, but how? any guide?

[prudhvigodithi]

[Untriage]

I see two issues reported here:

1. Too many default indices that come with the plugins, some of them are.opensearch-sap-log-types-config, .opensearch-sap-pre-packaged-rules-config, .plugins-ml-config.

2. Some default indices does not honor the discovery.type=single-node and cluster does not show up as green. (As mentioned in the comment the hack is to remove the plugins and delete the indices)

@dblock should @ryanqin01 start with plugins that create default indices to honor discovery.type=single-node and show green when cluster started with single node? and then circle back for point 1 dealing with too many default indices?

Also is the build repo right place for this issue? :), we can have a META issue created in Core repo and child issues in each repo to honor discovery.type=single-node.

Some helpful links with this topic:
#459
https://github.com/opensearch-project/ml-commons/blob/main/common/src/main/java/org/opensearch/ml/common/CommonValue.java#L61

[nilushancosta]

I am facing a somewhat related issue. I am trying to use the opensearch-operator to upgrade a cluster from 2.8.0 to 2.11.1. When a 2.11.1 pod starts, an .opensearch-sap-log-types-config index gets created. On a 3 node cluster, it requires 1 primary and 2 replica shards. So during the upgrade, the process gets stuck because all shards can't be allocated to a node.

[iqt4]

[Untriage]

I see two issues reported here:

1. Too many default indices that come with the plugins, some of them are.opensearch-sap-log-types-config, .opensearch-sap-pre-packaged-rules-config, .plugins-ml-config.
2. Some default indices does not honor the discovery.type=single-node and cluster does not show up as green. (As mentioned in the comment the hack is to remove the plugins and delete the indices)

@dblock should @ryanqin01 start with plugins that create default indices to honor discovery.type=single-node and show green when cluster started with single node? and then circle back for point 1 dealing with too many default indices?

Also is the build repo right place for this issue? :), we can have a META issue created in Core repo and child issues in each repo to honor discovery.type=single-node.

Some helpful links with this topic: opensearch-project/security-analytics#459 https://github.com/opensearch-project/ml-commons/blob/main/common/src/main/java/org/opensearch/ml/common/CommonValue.java#L61

Just came across this issue. On a fresh 2.12 installation I have these indices with unassigned shards
index shard prirep state
.opensearch-sap-pre-packaged-rules-config 0 r UNASSIGNED
.opendistro-ism-config 0 r UNASSIGNED
.opendistro-ism-managed-index-history-2024.06.02-1 0 r UNASSIGNED

[evheniyt]

Another workaround that doesn't require to delete any plugin.
Reduce the number of replicas for this index to 0:

/.opensearch-sap-log-types-config/_settings {"auto_expand_replicas":"false","index":{"number_of_replicas":0}}

Note:
this index is hidden, so you could search it with

/_cat/indices?expand_wildcards=all

[gaiksaya]

Moving this to security analytics plugin to take a look.
cc: @amsiglan

[dblock]

[Catch All Triage - 1, 2]

[tonytromp02]

I noticed that i could not simply remove the replica's and cluster stays yellow, for this .opensearch-sap-log-types-config index to become green i performed the following actions:

Disable the auto_expand_replicas

PUT /.opensearch-sap-log-types-config/_settings
{"index":{"auto_expand_replicas": null}}

Disable Replicas

PUT /.opensearch-sap-log-types-config/_settings
{"index":{"number_of_replicas":6}}