Wazuh-indexer testing: Initial start-up - Services, logs, templates, indexes and default configuration.
#2740
Assignees
Labels
Comments
2 tasks
Research on the
|
| Issue | Description |
|---|---|
| wazuh/wazuh-documentation#4999 | Additional steps need to be added to configure opensearch.yaml |
| wazuh/wazuh-packages#1413 | Fix the yml format of the config.yml file to generate the certificates |
| wazuh/wazuh-packages#1414 | Restarting the wazuh-indexer service remains blocked indefinitely. |
| wazuh/wazuh-packages#1749 | Warnings of wazuh-indexer confusing for the user. Mention in documentation |
Package installation 🟢
Details
The package has been installed following the installation guide and there have been no problems.
DEB
Run Commands:
# curl -s https://s3-us-west-1.amazonaws.com/packages-dev.wazuh.com/key/GPG-KEY-WAZUH | apt-key add -
# echo "deb https://s3-us-west-1.amazonaws.com/packages-dev.wazuh.com/pre-release/apt/ unstable main" | tee -a /etc/apt/sources.list.d/wazuh_pre_release.list
# apt install wazuh-indexer
Output
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
wazuh-indexer
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 357 MB of archives.
After this operation, 639 MB of additional disk space will be used.
Get:1 https://s3-us-west-1.amazonaws.com/packages-dev.wazuh.com/pre-release/apt unstable/main amd64 wazuh-indexer amd64 4.3.0-1 [357 MB]
Fetched 357 MB in 1min 20s (4,476 kB/s)
Selecting previously unselected package wazuh-indexer.
(Reading database ... 60956 files and directories currently installed.)
Preparing to unpack .../wazuh-indexer_4.3.0-1_amd64.deb ...
Creating wazuh-indexer group... OK
Creating wazuh-indexer user... OK
Unpacking wazuh-indexer (4.3.0-1) ...
Setting up wazuh-indexer (4.3.0-1) ...
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
Processing triggers for systemd (241-7~deb10u4) ...
RPM
Run Commands:
# rpm --import https://packages.wazuh.com/key/GPG-KEY-WAZUH
# echo -e '[wazuh_pre_release]\ngpgcheck=1\ngpgkey=https://s3-us-west-1.amazonaws.com/packages-dev.wazuh.com/key/GPG-KEY-WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://s3-us-west-1.amazonaws.com/packages-dev.wazuh.com/pre-release/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh_pre.repo
# yum install -y wazuh-indexer
Output
EL-8 - Wazuh 1.7 MB/s | 5.4 MB 00:03
Last metadata expiration check: 0:00:02 ago on Thu 31 Mar 2022 09:17:21 AM UTC.
Dependencies resolved.
========================================================================================================
Package Architecture Version Repository Size
========================================================================================================
Installing:
wazuh-indexer x86_64 4.3.0-1 wazuh_pre_release 361 M
Transaction Summary
========================================================================================================
Install 1 Package
Total download size: 361 M
Installed size: 614 M
Downloading Packages:
wazuh-indexer-4.3.0-1.x86_64.rpm 5.6 MB/s | 361 MB 01:04
--------------------------------------------------------------------------------------------------------
Total 5.6 MB/s | 361 MB 01:04
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Running scriptlet: wazuh-indexer-4.3.0-1.x86_64 1/1
Installing : wazuh-indexer-4.3.0-1.x86_64 1/1
Running scriptlet: wazuh-indexer-4.3.0-1.x86_64 1/1
Created opensearch keystore in /etc/wazuh-indexer/opensearch.keystore
Verifying : wazuh-indexer-4.3.0-1.x86_64 1/1
Installed:
wazuh-indexer-4.3.0-1.x86_64
Complete!
Cluster configuration 🔴
Details
It was not enough to follow the instructions in the documentation guide. Some additional steps are missing.
The following issue wazuh/wazuh-documentation#4999 has been opened to request to be added.
In addition, the following issue wazuh/wazuh-packages#1413 has been opened to report that the format of the file to generate the certificates config.yml does not have a correct yaml syntax.
Services 🔴
Details
When restarting the service with a configuration error in the /etc/wazuh-indexer/opensearch.yml file, the process hangs indefinitely. This has been reported in this issue wazuh/wazuh-packages#1414
Start 🟢
systemctl start wazuh-indexer
Output
root@wazuh-indexer-1:/home/vagrant# ps -aux | grep wazuh-indexer
wazuh-i+ 1550 14.4 65.7 3773184 1343448 ? Ssl 11:42 0:21 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-16082151826474642518 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy -XX:MaxDirectMemorySize=536870912 -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp /usr/share/wazuh-indexer/lib/* org.opensearch.bootstrap.OpenSearch -p /run/wazuh-indexer/wazuh-indexer.pid --quiet
root 1783 0.0 0.0 6208 880 pts/1 R+ 11:44 0:00 grep --color=auto wazuh-indexer
Stop 🟢
systemctl stop wazuh-indexer
Output
ps -aux | grep wazuh-indexer
root 1538 0.0 0.0 6208 884 pts/1 S+ 11:15 0:00 grep --color=auto wazuh-indexer
Normal Restart 🟢
systemctl restart wazuh-indexer
Output
root@wazuh-indexer-1:/home/vagrant# ps -aux | grep wazuh-indexer
wazuh-i+ 1550 14.4 65.7 3773184 1343448 ? Ssl 11:42 0:21 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/opensearch-16082151826474642518 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Dclk.tck=100 -Djdk.attach.allowAttachSelf=true -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy -XX:MaxDirectMemorySize=536870912 -Dopensearch.path.home=/usr/share/wazuh-indexer -Dopensearch.path.conf=/etc/wazuh-indexer -Dopensearch.distribution.type=rpm -Dopensearch.bundled_jdk=true -cp /usr/share/wazuh-indexer/lib/* org.opensearch.bootstrap.OpenSearch -p /run/wazuh-indexer/wazuh-indexer.pid --quiet
root 1783 0.0 0.0 6208 880 pts/1 R+ 11:44 0:00 grep --color=auto wazuh-indexe
Restart with configuration error 🔴
systemctl restart wazuh-indexer
Blocked indefinitely
Inital logs 🟡
Details
When starting the wazuh-indexer service, by default a series of warnings appear that can be confusing for the user. These warnings are inherited from opensearch.
Mar 30 11:12:12 qactl-ubuntu-1 systemd[1]: Starting Wazuh-indexer...
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: An illegal reflective access operation has occurred
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-de>
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Mar 30 11:12:29 qactl-ubuntu-1 systemd-entrypoint[669]: WARNING: All illegal access operations will be denied in a future release
Mar 30 11:13:01 qactl-ubuntu-1 systemd[1]: Started Wazuh-indexer.
This has been proposed to be referenced in the documentation through the issue wazuh/wazuh-packages#1749.
Templates and indexes 🟢
Details
After installing wazuh-indexer we proceeded to query the indexes with the following command:
curl -k -u admin:admin https://172.16.1.41:9200/_cat/indices?v
Output:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .opendistro_security S54wV-ShSt2EBAHngMdSxg 1 1 9 0 120.8kb 60.4kb
As you can see, by default the .opendistro_security index is created.
To test the correct functioning of the Filebeat templates and the indexing in wazuh-indexer, we proceeded to install Wazuh and Filebeat in another node, and connect it directly with our wazuh-indexer node.
After doing this and generating some example alerts, the indexes were queried again, and the following were obtained:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open wazuh-alerts-4.x-2022.03.31 KcQZ6MRSRJmGmDZplGuWpg 3 1 195 0 875.8kb 437.9kb
green open .opendistro_security S54wV-ShSt2EBAHngMdSxg 1 1 9 0 120.8kb 60.4kb
green open security-auditlog-2022.03.31 nu7dokAFRDWtKek59SYKsA 1 1 16 0 276.2kb 120.8kb
As can be seen, the Wazuh alerts have been indexed correctly, assuming that the Filebeat templates are correct.
Finally, we have visualized the index data of wazuh-alerts-4.x-2022.03.31 and obtained the following:
curl -k -u admin:admin https://172.16.1.41:9200/wazuh-alerts-4.x-2022.03.31/_search
{"took":83,"timed_out":false,"_shards":{"total":3,"successful":3,"skipped":0,"failed":0},"hits":{"total":{"value":196,"relation":"eq"},"max_score":1.0,"hits":[ ....]}
Default configuration 🟢
Details
The default opensearch.yml configuration template appears to be correct. What has been reported in wazuh/wazuh-documentation#4999 is that in the documentation it is necessary to specify some additional changes to those mentioned for the correct configuration of opensearch.yml
The following are some differences from the elasticsearch.yml file in 4.2.6.
Same sections between opensearch.yml and elasticsearch.yml
network.host
node.name
cluster.name
cluster.initial_master_nodes
discovery.seed_hosts
path.data
path.logs
Settings appearing in elasticsearch.yml and not in opensearch.yml
node.master
node.data
node.ingest
Settings appearing in opensearch.yml and not in elasticsearch.yml
compatibility.override_main_response_version # Option to allow Filebeat-oss 7.10.2 to work
Differences between plugin directives. plugins.security and opendistro_security
opensearch.yml
plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.audit.type: internal_opensearch
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
elasticsearch.yml
opendistro_security.ssl.transport.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch.pem
opendistro_security.ssl.transport.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: /etc/elasticsearch/certs/elasticsearch.pem
opendistro_security.ssl.http.pemkey_filepath: /etc/elasticsearch/certs/elasticsearch-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: /etc/elasticsearch/certs/root-ca.pem
opendistro_security.nodes_dn:
- CN=node-1,OU=Docu,O=Wazuh,L=California,C=US
- CN=node-2,OU=Docu,O=Wazuh,L=California,C=US
- CN=node-3,OU=Docu,O=Wazuh,L=California,C=US
opendistro_security.authcz.admin_dn:
- CN=admin,OU=Docu,O=Wazuh,L=California,C=US
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It is requested to make a review of the correct functioning after the installation of the
Wazuh-indexer. To do so, it is requested to verify if there is any anomalous behavior in:The text was updated successfully, but these errors were encountered: