Add Vulnerability Detector System End-to-End syscollector initial scans tests

[Deblintrake09]

Target version	Related issue	Related PR/dev branch
4.8.0	#4369	#4703

Description

This issue aims to create the basic End-to-end test cases defined in #4531. As all tests defined in the issue are part of the minimal required testing suite, and as tests have tight dependencies, the cases that are part of the basis for all test cases will be done in this issue, so development can be parallelized.

Proposed test cases

• E2E-VD-1: First syscollector scan (Rsync)
• E2E-VD-2: Subsequent scan (Dbsync) without any installation
  ...

Considerations

• This development is the basis for the rest of the test cases. It should be prioritized and developed ASAP to allow parallelization of development the rest of the tests.
• Tests must work on all proposed OSs.

[Rebits]

Found some provision errors:

• Windows 11 AMI does not include any python. This could lead into potential errors due to it is set by default a python interpreter that does actually not exist in the host. In addition, after some testing in the environment it seems that it is not possible to use some system testing tool over this Windows 11 hosts.
• macOS Agent is not correctly provisioned. More information here

It has been created a basic test structure and some fixture for configure the environment.
The current E2E approach has several limitations:

• Environment validation does not work for the Vulnerability Detector proposed environment. This should be adapted or disabled by default
• Configuration of the environment is implemented appending code block in the ossec.conf. It is difficult to implent complex configurations.

Created class method to configure the environment through YAML files

[Rebits]

Continue creating common fixtures for the basic tests.

[Rebits]

Created first PoC of the E2E-VD-3 tests and common structure for all the basic tests cases

[Rebits]

Development branch
enhacement/4590-vd-basic-test-cases

There still some issues regarding the monitoring of the logs. Currently working on it

[Rebits]

Development branch
enhacement/4590-vd-basic-test-cases

The Installation of a vulnerable package case was implemented correctly. This was performed following this case structure

- case: "Updating a vulnerable package that remains vulnerable to the same CVE"
  id: "update_vuln_package_remain_vulnerable"
  description: "Updating a vulnerable package that remains vulnerable to the same CVE"
  preconditions:
      tasks:
        - install_package:
            package:
              centos: https://downloads.rclone.org/v1.49.5/rclone-v1.49.5-linux-amd64.rpm
              ubuntu: https://downloads.rclone.org/v1.49.5/rclone-v1.49.5-linux-amd64.deb
      check_alerts:
          centos:
            - event: install_package
              name: "rclone"
              version: "1.49.5"
          ubuntu:
              - event: install_package
                name: "rclone"
                version: "1.49.5"
  body:
    tasks:
      - install_package:
          package:
            centos: https://downloads.rclone.org/v1.50.0/rclone-v1.50.0-linux-amd64.rpm
            ubuntu: https://downloads.rclone.org/v1.50.0/rclone-v1.50.0-linux-amd64.deb
    check_alerts:
        centos:
          - event: upgrade_package
            name: "rclone"
            version: "1.50.0"
        ubuntu:
            - event: upgrade_package
              name: "rclone"
              version: "1.50.0"
  teardown:
    - remove_package:
        package:
          centos: rclone
          ubuntu: rclone

This approach will allow us to include desired cases easily. However this first iteration do not include dependency.
The implementation of dynamic dependency could lead into delays in the development

Warning
Currently it is only checked that the package installation/removal/upgrade alert was generated in the manager. It would be necessary a development branch to implement a fully functional tests.

Currently working

• Support for Windows Endpoint
• Implementation in the rest of the tests cases
• Creation of a solid vulnerability packages repository

[Rebits]

• Refactor tests structure in order to improve readability
• Fix Windows endpoint
• Include support for ARM endpoints

[Rebits]

• Basic Syscollector scan are stable for all the endpoints (AMD architectures)
• Refactor monitoring methods and improve readability
• Fix framework error for windows package installation

[Rebits]

• Fixed minor errors in installation of packages for windows and macOS systems
• Fixed minor errors handling different architectures cases
• Refactor tests in order to migrate some monitoring logic to fixtures
• Working in implementing Vulnerabilities monitoring through API and alerts

[Rebits]

• Working in apply dependency for each tests cases
• Including indexer alerts logic
• Include API method to check state index
• Refactor some fixtures

[Rebits]

• Meeting with @pereyra-m and @Deblintrake09 in order to clarify some of the preconditions specified in Determine VDT System tests tiers, preconditions, implications and expectations #4531
  • Tests should only wait for Syscollector Scans. Alerts should be generated and indexes created intermediately after (Vulnerability Scan interval is deprecated).
  • State indices will match https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html

Regarding this clarifications it is necessary to:

• Remove time wait between Syscollector scan in all agents and vulnerability alerts monitoring for the agent
• Include methods to gather information from new state indices

---

Currently working in tests dependency and fixing minor errors in vulnerability alerts gathering. Changes here: 21176da

New ETA 17/11/23 due the following reasons:

• Tests dependency did not work as expected. We are currently working on Add Vulnerability Detector System End-to-End syscollector initial scans tests #4590 (comment) schema to dynamically address test case dependency
• ARM hosts package management and monitoring has not been tested yet

[Rebits]

Refactor VD tests in order to improve migration to new framework. Done in b9f1101
Also refactor schema in order to check generated vulnerabilities in different ways.

[Rebits]

Created PR with development #4703
It is required further testing

Note

Current approach implement host handling functionality using different E2E modules. This is a workaround in order to avoid the migration of the QA framework. However it is recommended to follow a class approach to implement those operations.

---

During testing it has been detected a possible issue handling Vulnerabilities of Windows agents registered in worker nodes. Currently researching this issue.
In addition it has been detected that HostMonitor approach for monitoring the alerts generated in the environment is not viable due to the large amount of alerts produced in a multi agent environment.
It is suggested to check alerts performing API requests to the indexer server instead.

[Rebits]

• Include docstrings and refactor Syscollector initial scans
• Include logger for VD tests
• Fix errors in provisioning of filebeat and E2E environemnts

---

Currently testing in real environment
It seems that in real testing Vulnerability Scan of agents is waiting longer that the expected: https://ci.wazuh.info/job/Test_e2e_system/186/console
It is necessary to analyze result. In addition, it is necessary to include a minor logic to gather information about the environment

[Rebits]

• Fixed enrollment configuration error in the agents
• Currently working in solving timeout errors:

Testing build: https://ci.wazuh.info/job/Test_e2e_system/189/console

[Rebits]

Working on PR review requested changes

[Rebits]

Moved ETA to 27/11/2023 to apply the rest of the PR comments from @juliamagan

[Rebits]

• Apply PR requested changes by @juliamagan

[Rebits]

Moved ETA to 28/11/2023 to allow the revision from @juliamagan