`Wazuh-indexer` testing: User experience - Usability when generating certificates and configuring the cluster. #2738

jmv74211 · 2022-03-29T10:41:09Z

Parent issue
wazuh/wazuh#12901

We want to test the usability and user experience when generating the necessary certificates for encrypted communication between components and cluster configuration.

The process must be clear and simple according to the documentation indicated. Any kind of problem, inconvenience, or suggestion should be reported.

Note: All research and results obtained should be attached in comments to this issue.

fedepacher · 2022-03-30T21:28:01Z

Research into `wazuh-indexer`: User experience

Reported issues

Issue	Description	Severity
wazuh/wazuh#12934	Problem launching wazuh-certs-tool.sh script	High
wazuh/wazuh-documentation#5008	Missing information in the documentation when configuring Wazuh-Indexer	High
wazuh/wazuh-packages#1749	Warnings messages when checking Wazuh-Indexer status	Low
wazuh/wazuh-packages#1885	Infinite loop when running indexer-security-init.sh	High
wazuh/wazuh-documentation#4999	Missing information in the installation process of `filebeat`	Low
wazuh/wazuh#12973	Obsolete variable name in `opensearch_dashboards.yml` configuration file	Low
wazuh/wazuh-dashboard-plugins#3941	Permission error in `wazuh-dashboard` logs when running for first time	Low

Installing the `wazuh-indexer` in step-by-step mode 🔴

Details

Certificates creation 🔴

Details

Download the wazuh-cert-tool.sh script and the config.yml configuration file. 🟢
Edit ./config.yml and replace the node names and IP values 🟢
Run the ./wazuh-certs-tool.sh to create the certificates 🔴
Compress all the necessary files 🟢
Copy wazuh-certificates.tar to all the nodes 🟢

Problem:

[root@centos]# bash ./wazuh-certs-tool.sh -A
29/03/2022 13:33:42 INFO: Admin certificates created.	
29/03/2022 13:33:42 ERROR: The given information does not match with an IP address or a DNS.

The following issue https://github.com/wazuh/wazuh/issues/12934 has been opened to request to be fixed.

Note: This problem is caused by the vi editor, the CRLF must be replaced by LF.

Solution:
I used nano and I replaced the CRLF by the LF. 🟢

[root@centos]# bash ./wazuh-certs-tool.sh -A
30/03/2022 18:04:42 INFO: Admin certificates created.
30/03/2022 18:04:43 INFO: Wazuh indexer certificates created.
30/03/2022 18:04:43 INFO: Wazuh server certificates created.
30/03/2022 18:04:43 INFO: Wazuh dashboard certificates created.

Node installation

Installing package dependencies 🟢

Details

RPM: 🟢
DEB: 🟢

Adding the Wazuh repository 🟢

Details

RPM: 🟢
DEB: 🟢

Installing the `wazuh-indexer` 🟢

Details

RPM: 🟢
DEB: 🟢

Configuring the `wazuh-indexer` 🟡

Details

Missing information in the documention about /etc/wazuh-indexer/opensearch.yml file. 🟡
Deploying certificates 🟢

The following issue wazuh/wazuh#12947 has been opened to request to be fixed.

Starting the service 🟡

Details

systemctl status wazuh-indexer 🟡

Result:

systemctl status wazuh-indexer
Mar 30 11:12:12 qactl-ubuntu-1 systemd[1]: Starting Wazuh-indexer...
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: An illegal reflective access operation has occurred
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-de>
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: All illegal access operations will be denied in a future release
Mar 30 11:13:01 qactl-ubuntu-1 systemd[1]: Started Wazuh-indexer.

Note: The documentation should justify the warning messages.

The following issue wazuh/wazuh-packages#1749 has been opened to request to be fixed.

Cluster initialization 🔴

Details

Run the wazuh-indexer indexer-security-init.sh script on any wazuh-indexer node to load the new certificates information and start the cluster. 🟡

Note: if it is well configured the file /etc/wazuh-indexer/opensearch.yml the test of the cluster get 🟢

The following issue wazuh/wazuh-packages#1885 has been opened to request to be fixed.

Result:

[root@qactl-centos-8-1 vagrant]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Security Admin v7
Will connect to 172.18.1.10:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: GREEN
Number of nodes: 3
Number of data nodes: 3
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/
Will update '_doc/config' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/config.yml 
   SUCC: Configuration for 'config' created or updated
Will update '_doc/roles' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '_doc/rolesmapping' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '_doc/internalusers' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
Will update '_doc/actiongroups' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/action_groups.yml 
   SUCC: Configuration for 'actiongroups' created or updated
Will update '_doc/tenants' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
Will update '_doc/nodesdn' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/nodes_dn.yml 
   SUCC: Configuration for 'nodesdn' created or updated
Will update '_doc/whitelist' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/whitelist.yml 
   SUCC: Configuration for 'whitelist' created or updated
Will update '_doc/audit' with /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/audit.yml 
   SUCC: Configuration for 'audit' created or updated
Done with success

Note: if the node is not well configured (/etc/wazuh-indexer/opensearch.yml) and run the command /usr/share/wazuh-indexer/bin/indexer-security-init.sh, it will state in an infinite loop trying to connect, I have to reboot the VM. 🔴

Result:

[root@qactl-centos-8-1 vagrant]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Testing the cluster installation 🟢

Details

Result:

[root@qactl-centos-8-1 vagrant]# curl -k -u admin:admin https://172.18.1.10:9200
{
  "name" : "node-1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "223vQw6IQda5Ui6HXwKyWQ",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
    "build_date" : "2022-01-14T03:38:06.881862Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[root@qactl-centos-8-1 vagrant]# curl -k -u admin:admin https://172.18.1.10:9200/_cat/nodes?v
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.18.1.10           37          48   0    0.01    0.01     0.00 dimr      *      node-1
172.18.1.11           43          93   0    0.00    0.00     0.00 dimr      -      node-2
172.18.1.12           68          63   0    0.00    0.01     0.00 dimr      -      node-3

Installing the `wazuh-server` in step-by-step mode 🟡

Details

`wazuh-server` node installation

Adding the Wazuh repository 🟢

Details

RPM: 🟢
DEB: 🟢

Installing the Wazuh manager 🟢

Details

RPM: 🟢
DEB: 🟢

Installing Filebeat 🟢

Details

RPM: 🟢
DEB: 🟢

Configuring Filebeat 🟢

Details

Download the preconfigured Filebeat configuration file. 🟢
Edit the /etc/filebeat/filebeat.yml configuration file. 🟢
Create a secrets keystore. 🟢
Add the username and password using the following commands. 🟢
Download the alerts template for the wazuh-indexer. 🟢
Download the Wazuh module. 🟢
Deploying certificates. 🟢

Starting the Filebeat service 🟡

Details

Enable and start the Filebeat service. 🟢
Run the following command to verify that Filebeat is successfully installed. 🟡

The following comment in the issue wazuh/wazuh#12878 (comment) has been made.

Solution:

In the wazuh-indexer node:

stop firewalld
flush iptable (iptable -F)
reboot the system

Result:

[root@qactl-centos-8-4 vagrant]# filebeat test output
elasticsearch: https://172.18.1.10:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.18.1.10
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
elasticsearch: https://172.18.1.11:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.18.1.11
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2
elasticsearch: https://172.18.1.12:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.18.1.12
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... OK
  version: 7.10.2

Cluster configuration for multi-node deployment 🟢

Details

Configuring the wazuh-server master node in /var/ossec/etc/ossec.conf file. 🟢
Configuring the wazuh-server worker node in /var/ossec/etc/ossec.conf file. 🟢
Testing wazuh-server cluster. 🟢

Result:

[root@qactl-centos-8-4 vagrant]# /var/ossec/bin/cluster_control -l
NAME     TYPE    VERSION  ADDRESS      
wazuh-1  master  4.3.0    172.18.1.13  
wazuh-2  worker  4.3.0    172.18.1.14

Installing the `wazuh-dashboard` in step-by-step mode 🟡

Details

`wazuh-dashboard` installation

Installing package dependencies 🟢

Details

RPM: 🟢
DEB: 🟢

Adding the Wazuh repository 🟢

Details

RPM: 🟢
DEB: 🟢

Installing the `wazuh-dashboard` 🟢

Details

RPM: 🟢
DEB: 🟢

Configuring the `wazuh-dashboard` 🟡

Details

Edit the /etc/wazuh-dashboard/opensearch_dashboards.yml file. 🟢

Note: This file make reference to a field with the name kibana_read_only. It is an obsolete name.

Deploying certificates. 🟢
Starting the Wazuh dashboard service. 🟡

The following issue wazuh/wazuh#12973 has been opened to check the name that make reference to Kibana.

Result:

● wazuh-dashboard.service - wazuh-dashboard
   Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-03-30 20:19:19 UTC; 14min ago
 Main PID: 49101 (node)
    Tasks: 11 (limit: 24918)
   Memory: 136.1M
   CGroup: /system.slice/wazuh-dashboard.service
           └─49101 /usr/share/wazuh-dashboard/bin/../node/bin/node /usr/share/wazuh-dashboard/bin/../src/cli/dist -c /etc/wazuh-dashboard/opensearch_dashboards.yml

Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["info","plugins-system"],"pid":49101,"message":"Starting [45] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,repo>
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["error","opensearch","data"],"pid":49101,"message":"[ResponseError]: Response Error"}
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["error","plugins","wazuh","initialize"],"pid":49101,"message":"Could not check if the index .wazuh exists due to no permissions for create, delete or check"}
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["error","opensearch","data"],"pid":49101,"message":"[ResponseError]: Response Error"}
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["listening","info"],"pid":49101,"message":"Server running at https://172.18.1.16:443"}
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["info","http","server","OpenSearchDashboards"],"pid":49101,"message":"http server running at https://172.18.1.16:443"}
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["error","opensearch","data"],"pid":49101,"message":"[ResponseError]: Response Error"}
Mar 30 20:19:25 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:19:25Z","tags":["error","opensearch","data"],"pid":49101,"message":"[ResponseError]: Response Error"}
Mar 30 20:30:01 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:30:01Z","tags":["error","opensearch","data"],"pid":49101,"message":"[resource_already_exists_exception]: index [wazuh-monitoring-2022.13w/FtklHG83TeWBepTLyurjMw] already exists"}
Mar 30 20:30:01 qactl-centos-8-7 opensearch-dashboards[49101]: {"type":"log","@timestamp":"2022-03-30T20:30:01Z","tags":["error","plugins","wazuh","monitoring"],"pid":49101,"message":"Could not create wazuh-monitoring-2022.13w index on elasticsearch due to resource_already_exists_exception"}

Also, the /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log file shows the error while checking the index. The following issue wazuh/wazuh-dashboard#89 has been opened to request to be fixed.

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log
{"date":"2022-03-30T20:19:25.228Z","level":"info","location":"initialize","message":"Wazuh dashboard index: .kibana"}
{"date":"2022-03-30T20:19:25.228Z","level":"info","location":"initialize","message":"App revision: 4301-1"}
{"date":"2022-03-30T20:19:25.228Z","level":"info","location":"initialize","message":"Total RAM: 3930MB"}
{"date":"2022-03-30T20:19:25.410Z","level":"error","location":"initialize:checkKibanaStatus","message":"Could not check if the index .wazuh exists due to no permissions for create, delete or check"}
{"date":"2022-03-30T20:19:25.750Z","level":"error","location":"monitoring:getApiInfo","message":"connect ECONNREFUSED 127.0.0.1:55000"}
{"date":"2022-03-30T20:30:01.736Z","level":"error","location":"monitoring:createIndex","message":"Could not create wazuh-monitoring-2022.13w index on elasticsearch due to resource_already_exists_exception"}

Note: The file /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml won't be created if the wazuh-dashboard does not initilize correctly.

jmv74211 added team/qa manual-testing labels Mar 29, 2022

jmv74211 mentioned this issue Mar 29, 2022

Release 4.3.0 - Manual tests - Wazuh indexer wazuh/wazuh#12901

Closed

2 tasks

jmv74211 assigned fedepacher Mar 29, 2022

jmv74211 closed this as completed Apr 1, 2022

jsoliani mentioned this issue Apr 29, 2022

Release 4.3.0 - Wazuh-Indexer manual tests wazuh/wazuh#13273

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`Wazuh-indexer` testing: User experience - Usability when generating certificates and configuring the cluster. #2738

`Wazuh-indexer` testing: User experience - Usability when generating certificates and configuring the cluster. #2738

jmv74211 commented Mar 29, 2022 •

edited

Loading

fedepacher commented Mar 30, 2022 •

edited

Loading

Certificates creation 🔴

Node installation

Installing package dependencies 🟢

Adding the Wazuh repository 🟢

Installing the `wazuh-indexer` 🟢

Configuring the `wazuh-indexer` 🟡

Starting the service 🟡

Cluster initialization 🔴

Testing the cluster installation 🟢

`wazuh-server` node installation

Adding the Wazuh repository 🟢

Installing the Wazuh manager 🟢

Installing Filebeat 🟢

Configuring Filebeat 🟢

Starting the Filebeat service 🟡

Cluster configuration for multi-node deployment 🟢

`wazuh-dashboard` installation

Installing package dependencies 🟢

Adding the Wazuh repository 🟢

Installing the `wazuh-dashboard` 🟢

Configuring the `wazuh-dashboard` 🟡

Wazuh-indexer testing: User experience - Usability when generating certificates and configuring the cluster. #2738

Wazuh-indexer testing: User experience - Usability when generating certificates and configuring the cluster. #2738

Comments

jmv74211 commented Mar 29, 2022 • edited Loading

fedepacher commented Mar 30, 2022 • edited Loading

Research into wazuh-indexer: User experience

Reported issues

Installing the wazuh-indexer in step-by-step mode 🔴

Certificates creation 🔴

Node installation

Installing package dependencies 🟢

Adding the Wazuh repository 🟢

Installing the wazuh-indexer 🟢

Configuring the wazuh-indexer 🟡

Starting the service 🟡

Cluster initialization 🔴

Testing the cluster installation 🟢

Installing the wazuh-server in step-by-step mode 🟡

wazuh-server node installation

Adding the Wazuh repository 🟢

Installing the Wazuh manager 🟢

Installing Filebeat 🟢

Configuring Filebeat 🟢

Starting the Filebeat service 🟡

Cluster configuration for multi-node deployment 🟢

Installing the wazuh-dashboard in step-by-step mode 🟡

wazuh-dashboard installation

Installing package dependencies 🟢

Adding the Wazuh repository 🟢

Installing the wazuh-dashboard 🟢

Configuring the wazuh-dashboard 🟡

`Wazuh-indexer` testing: User experience - Usability when generating certificates and configuring the cluster. #2738

`Wazuh-indexer` testing: User experience - Usability when generating certificates and configuring the cluster. #2738

jmv74211 commented Mar 29, 2022 •

edited

Loading

fedepacher commented Mar 30, 2022 •

edited

Loading

Research into `wazuh-indexer`: User experience

Installing the `wazuh-indexer` in step-by-step mode 🔴

Installing the `wazuh-indexer` 🟢

Configuring the `wazuh-indexer` 🟡

Installing the `wazuh-server` in step-by-step mode 🟡

`wazuh-server` node installation

Installing the `wazuh-dashboard` in step-by-step mode 🟡

`wazuh-dashboard` installation

Installing the `wazuh-dashboard` 🟢

Configuring the `wazuh-dashboard` 🟡