Release 4.10.0 - RC 3 - Vulnerability Detection performance test

[jnasselle]

Tests information

Main release stage issue	#27524
Main Vulnerability Detection tests issue	#27530
Version	4.10.0
Release stage	RC 3
Tag	https://github.com/wazuh/wazuh/tree/v4.10.0-rc3
Previous Vulnerability performance tests issue	#27361

Description

The objective is to conduct performance tests to analyze the vulnerability detection module across varying environment loads: high, medium, and low. This comparative analysis will provide a comprehensive understanding of the current status of the vulnerability detection module and will help identify any unexpected behavior.

Methodology

Utilizing the CLUSTER-Workload_benchmarks_metrics pipeline to execute specified test cases automatically. Results will be manually analyzed and shared with the development team for validation adjustments.

Test Cases

Case	Description	Workers	Indexers	Agents	EPS	Time
High Activity	Simulate a large-scale environment with significant activity	2	2	200	50	3h

Note

Normal and Very High activity cases will not be executed in this RC, as approved by @juliamagan

Test parameters

Parameters

• TARGET_PKG_VERSION: 4.10.0
• TARGET_PKG_REVISION: 1
• TARGET_REPOSITORY: pre-release
• VERBOSITY: -v
• SOURCE_REFERENCE: v4.10.0-rc3
• JENKINS_REFERENCE: v4.10.0-rc3
• QA_REFERENCE: v4.10.0-rc3
• TEST_TIME: (As specified in test cases: 3h -> 10800, 1h -> 3600)
• LOAD_BALANCER: elb
• PROTOCOL: TCP
• OS: debian10
• AGENT_SIMULATED_VERSION: 4.10.0
• LABELS: left empty
• EVENTS_SIZE: 0
• MODE: HYBRID
• AGENT_NUMBER: (As specified in test cases)
• MANAGER_WORKERS: (As specified in test cases)
• INDEXER_NODES: (As specified in test cases)
• DASHBOARD: True
• PARSE_CLUSTER_LOGS: False
• CLUSTER_TESTS: left empty
• KIBANA_API_REQUESTS: True
• TEST_PERFORMANCE_API_ENDPOINTS: False
• EXTRA_LOAD_API_REQUESTS: True
• UNIFY_BINARY_PROCESSES_FOR_PLOTTING: True
• API_REQUESTS_PERCENTAGE: 10
• MANAGERS_AWS_DEPLOYMENT: EC2
• MANAGER_INSTANCES: auto
• AGENT_INSTANCES: auto
• QUEUES: Vulnerability
• FIM_EPS: 0
• ROOTCHECK_EPS: 0
• SYSCOLLECTOR_EPS: 0
• VULNERABILITY_EPS: (Twice the number specified in the test cases)
• LOGCOLLECTOR_EPS: 0
• SCA_EPS: 0
• WINDOWS_EVENT_EPS: 0
• HOST_INFO_EPS: 0
• FIM_INTEGRITY_EPS: 0
• MANAGER_MONITORING_PROCESSES: wazuh-analysisd,wazuh-remoted,wazuh-db,wazuh-modulesd,wazuh-apid,wazuh_clusterd,wazuh-authd,wazuh-dbd
• MANAGER_MONITORING_STATS: remote,analysis,wazuhdb
• INDEXER_MONITORING: wazuh-indexer
• INDEXER_MONITORING_STATS: vulnerabilities,alerts
• DASHBOARD_MONITORING: wazuh-dashboard
• LOGGING: alerts.log
• GROUPS_NUMBER: 0
• FILES_NUMBER_PER_GROUP: 0
• BYTES_NUMBER_PER_FILE: 0
• LOCAL_INTERNAL_OPTIONS_CONFIG: monitord.rotate_log=0,wazuh_database.max_queued_events=33554431,wazuh_clusterd.debug=1 (each , is a new line)
• DESTROY_INSTANCES: True

Test considerations

• Test frequency: 60
• Number of vulnerable packages: 100

---

Builds and artifacts

• High: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/838/
  • Artifacts: artifacts.zip

Conclusion 🔴

• New Issues:

  • Some components from Worker 2 stopped working during VD Performance test #27547
  • Vulnerability scanner error in performance tests #27546

• Known issues:

  • Agent not found error in cluster performance tests #8480
  • [WARNING] Fail to read queue capacity via reflection warning in Indexer log wazuh-indexer#71
  • [BUG] Error during startup Failed loading builtin log types from disk! opensearch-project/security-analytics#1312
  • Huge memory increase of wazuh-modulesd in Vulnerability Detection performance test #27260

[Rebits]

• Waiting for the tests to complete: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/838/

[MARCOSD4]

High Activity

Comparison: #26172

Build: https://ci.wazuh.info/job/CLUSTER-Workload_benchmarks_metrics/838/

Artifacts:

• artifacts.zip

Logs 🔴

Summary

Master 🟡

• Expected:

2025/01/08 16:24:12 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2025/01/08 16:24:12 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-x-x-x-x.ec2.internal', retrying until the connection is successful.
2025/01/08 16:33:20 indexer-connector: WARNING: No username and password found in the keystore, using default values.

• Not Know issue: Agent not found error in cluster performance tests #8480

2025/01/08 16:38:24 wazuh-remoted: ERROR: (1320): Agent '000' not found.

Worker 1 🟢

• Expected warnings:

2025/01/08 16:24:11 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2025/01/08 16:24:11 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-x-x-x-x.ec2.internal', retrying until the connection is successful.
2025/01/08 16:33:21 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2025/01/08 16:39:49 wazuh-analysisd: WARNING: Syscollector decoder queue is full.
2025/01/08 16:40:30 wazuh-remoted: WARNING: Message queue is full (131072). Events may be lost.

Worker 2 🔴

• Expected errors:

2025/01/08 16:24:10 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2025/01/08 16:24:10 indexer-connector: WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-ip-x-x-x-x.ec2.internal', retrying until the connection is successful.
2025/01/08 16:33:21 indexer-connector: WARNING: No username and password found in the keystore, using default values.
2025/01/08 16:39:37 wazuh-remoted: WARNING: Package dropped. Could not append data into buffer.
2025/01/08 16:39:37 wazuh-remoted: WARNING: (1246): Unable to send file 'merged.mg' to agent ID '083'.
2025/01/08 16:39:45 wazuh-analysisd: WARNING: Syscollector decoder queue is full.

• New issue: Vulnerability scanner error in performance tests #27546

2025/01/08 19:22:41 wazuh-modulesd:vulnerability-scanner: ERROR: VulnerabilityScannerFacade::initEventDispatcher: Failed to enqueue element: 00000000000040508691 - Event message: �

Indexer 0 🟡

• Expected opensearch warnings:

[2025-01-08T16:20:38,905][WARN ][stderr                   ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-01-08T16:20:38,905][WARN ][stderr                   ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-01-08T16:20:38,905][WARN ][stderr                   ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-01-08T16:20:42,014][WARN ][o.o.s.c.Salt             ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2025-01-08T16:20:42,053][ERROR][o.o.s.a.s.SinkProvider   ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] Default endpoint could not be created, auditlog will not work properly.
[2025-01-08T16:20:42,054][WARN ][o.o.s.a.r.AuditMessageRouter] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] No default storage available, audit log may not work properly. Please check configuration.
[2025-01-08T16:20:42,918][WARN ][o.o.s.p.SQLPlugin        ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2025-01-08T16:20:43,379][ERROR][o.o.p.c.j.GCMetrics      ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] MX bean missing: G1 Concurrent GC
[2025-01-08T16:20:43,380][WARN ][o.o.p.c.ThreadPoolMetricsCollector] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] Fail to read queue capacity via reflection
[2025-01-08T16:20:44,387][WARN ][o.o.g.DanglingIndicesState] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-01-08T16:20:45,229][ERROR][o.o.s.l.BuiltinLogTypeLoader] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] Failed loading builtin log types from disk!
[2025-01-08T16:20:46,216][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] Config override setting update called with empty string. Ignoring.
[2025-01-08T16:20:46,943][WARN ][o.o.o.i.ObservabilityIndex] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] message: index [.opensearch-observability/szxpUUSVQRqW6oSKcX21NQ] already exists
[2025-01-08T16:23:25,967][WARN ][o.o.s.c.ConfigurationRepository] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] Unable to reload configuration, initalization thread has not yet completed.

Known Issue: wazuh/wazuh-indexer#71

[2025-01-08T16:20:35,553][INFO ][o.o.n.Node               ] [CLUSTER-Workload_benchmarks_metrics_B838_indexer_0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]

Known Issue: opensearch-project/security-analytics#1312

at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.16.0.jar:2.16.0]

Indexer 1 ⚪

No Logs

Dashboard ⚪

• No Logs

Metrics and Statistics 🔴

Master 🟢

• Metrics:

  • Disk_read, Disk_written, FD, PSS, RSS, USS: Decrease in wazuh-modulesd, probably related to the fix introduced in Huge memory increase of wazuh-modulesd in Vulnerability Detection performance test #27260.

• Statistics:

  • No abnormalities found

---

• Binaries

• Stats: wazuhdbd

• Stats: remoted

• Stats: analysisd

Dashboard 🟢

• Metrics

No abnormalities were found.

• Statistics

No abnormalities were found.

---

• Binaries

Worker 1 🟢

• Metrics:

  • Disk_read, Disk_written, FD: decrease in wazuh-modulesd, probably related to the fix introduced in Huge memory increase of wazuh-modulesd in Vulnerability Detection performance test #27260

• Statistics:
  No abnormalities were found.

---

• Binaries

• Stats: wazuhdbd

• Stats: remote

• Stats: analysis

Worker 2 🔴

• Metrics:

  • Some components of Worker 2 stopped working. Opened Issue: Some components from Worker 2 stopped working during VD Performance test #27547

• Statistics:

  • No abnormalities found.

---

• Binaries

• Stats: wazuhdbd

• Stats: remoted

• Stats: analysis
  
  
  
  

Indexer 1 🟢

• Metrics

  • Small increase in RSS, PSS, USS.

• Statistics

  • No abnormalities found.

---

• Binaries
  
  
  
  
  
  
  
  
  
  
  
  
  

• Stats
  
  
  

Indexer 2 🟢

• Metrics

  • Small increase in RSS, PSS, USS.

• Statistics

  • No abnormalities found.

---

• Binaries
  
  
  
  
  
  
  
  
  
  
  
  
  

• Stats
  
  
  

Vulnerabilities State 🟢

Alerts 🟢

[Rebits]

Regarding reported issues from the test evidence perspective

• Vulnerability scanner error in performance tests #27546
• Some components from Worker 2 stopped working during VD Performance test #27547

We can confirm the following facts:

• Remoted went down at approximately 2025/01/08 19:21:00, coinciding with the reported error timestamp (refer to Vulnerability scanner error in performance tests #27546). The services remained unavailable for the remainder of the test

Simulate agent logs

INFO:P24:{'keepalive': {'status': 'enabled', 'frequency': 10.0}, 'fim': {'status': 'disabled', 'eps': 0}, 'fim_integrity': {'status': 'disabled', 'eps': 0}, 'syscollector': {'status': 'disabled', 'frequency': 60, 'eps': 0}, 'vulnerability': {'status': 'enabled', 'frequency': 60, 'eps': 100}, 'rootcheck': {'status': 'disabled', 'frequency': 60.0, 'eps': 0}, 'sca': {'status': 'disabled', 'frequency': 60, 'eps': 0}, 'hostinfo': {'status': 'disabled', 'eps': 0}, 'winevt': {'status': 'disabled', 'eps': 0}, 'logcollector': {'status': 'disabled', 'eps': 0}, 'receive_messages': {'status': 'enabled'}}
INFO:P24:Waiting 30 seconds before sending EPS and keep-alive events
INFO:P24:Starting 200 agents.
WARNING:root:Broken Pipe error while sending event. Creating new socket...
WARNING:root:Broken Pipe error while sending event. Creating new socket...
WARNING:root:Broken Pipe error while sending event. Creating new socket...
...
Exception in thread Thread-032vulnerability:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/wazuh_testing-4.10.0-py3.9.egg/wazuh_testing/tools/agent_simulator.py", line 1791, in send_event
    self.socket.send(length + event)
BrokenPipeError: [Errno 32] Broken pipe
...

• Binary and stats data for serveral daemons stops at 2025/01/08 19:21:00 approximately
  • wazuh-modulesd -> Last stats collected at 19:22:19
  • wazuh-remoted -> Last stats collected at 19:29:31
  • wazuh-authd -> Last stats collected at 19:29:15
  • wazuh-cluster -> Showed in plots as wazuh-c, further research is required here. -> Last stats collected 19:29:47
  • wazuh-analysisd -> Last stats collected at 19:28:41

However several daemons were working correctly at that time. For example, worker2's last cluster log was at 19:44:18

2025/01/08 19:44:18 INFO: [Worker CLUSTER-Workload_benchmarks_metrics_B838_manager_2] [Integrity check] Finished in 0.008s. Sync is not required.

In addition, it seems like the csv was truncated somehow. For example, the remoted file's last lines are:

wazuh-remoted,4.10.0,2025/01/08 19:29:26,29812,147.6,1293276.0,73796.0,67920.0,69164.0,0.0,210,847101,631312,3304.0,117760.0,0.0,140.0
wazuh-remoted,4.10.0,2025/01/08 19:29:31,29812,149.1,1293276.0,73796.0,67920.0,69164.0,0.0,210,847584,631447,3304.0,118296.0,%                                                                

The final values for the csv are missing. This could be related to a lack of disk space in the environment

[santipadilla]

LGTM

[juliamagan]

A new issue has been created to update the test: https://github.com/wazuh/wazuh-qa-automation/issues/673