deploy: d3a7a36

README.md

@@ -0,0 +1,24 @@
+<!--
+SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# OSFV documentation README
+
+Introducing new platform to the Open Source Firmware Validation infrastructure
+requires acquiring crucial parameters of the Device Under Tests (DUT). To
+leverage this process, users can refer to the following diagrams and answer the
+accompanying questions.
+
+## Power State requirements
+
+![power-state](./img/power-state-v0.1.0.svg)
+
+## Power Supply requirements
+
+![power-supply](./img/power-supply-v0.1.0.svg)
+
+## Serial requirements
+
+![serial](./img/serial-v0.1.0.svg)

adding-new-platforms.md

@@ -0,0 +1,123 @@
+<!--
+SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Adding new platforms
+
+Depending on what type of platform you're adding, the instructions here will
+vary.
+
+- If no similar board is yet supported, follow the steps in
+  [Adding a brand new platform](#adding-a-brand-new-platform)
+- If the board is a variant of another, similar, already supported board, follow
+  the steps in
+  [Adding new variant of an existing platform](#adding-new-variant-of-an-existing-platform)
+
+## Generating config variables
+
+To simplify filling in variables in config, you can use
+`scripts/get-robot-variables.sh` on the target device. Remember to fill in the
+hardware slots (such as WiFi) with the peripherals to be used during tests. It
+is assumed that this script is executed on the Ubuntu OS. You can use these
+automatically generated variables as another input to the cases described below.
+
+Also, if there already is a Dasharo FW binary for the platform, you
+might want to extract the `.config` from the binary and try to run it through
+our [config parser](config-parser.md) to get at least some flags automatically.
+
+## Adding a brand new platform
+
+- Create a new file for your mainboard in `platform-configs/`. For most
+  platforms this file will be called `[platform-vendor]-[platform-model].robot`.
+- Copy the contents of `include/default.robot` to your platform config
+- Modify the file for your platform:
+    + Modify the settings appropriately for your mainboard
+    + Remove any unmodified lines - they will be sourced from `default.robot`
+    + Add the following at the top of your platform config - this will ensure
+      defaults are used for unspecified options:
+
+    ```robot
+    *** Settings ***
+    Resource    default.robot
+    ```
+
+- Add the platform configuration to `variables.robot:
+    + Create a new configuration of RTE, if you are using one, e.g.:
+
+        ```robot
+        &{RTE11}=                   ip=192.168.10.174
+        ...                         platform=apu4    platform_vendor=PC Engines
+        ```
+
+    + Add the RTE to the list:
+
+        ```robot
+        @{RTE_LIST}=                &{RTE05}
+        ...                         &{RTE06}    &{RTE07}    &{RTE08}    &{RTE09}    &{RTE10}
+        ...                         &{RTE11}
+        ```
+
+    + Do the same for any modules installed in the platform
+    + Create a new CONFIG containing the RTE and modules used for testing, and
+      append it to the list:
+
+        ```robot
+        @{CONFIG04}=                &{RTE11}    &{SSD06}    &{CARD06}    &{USB03}
+        ...                         &{MODULE06}    &{ADAPTER01}    &{MODULE10}
+
+        @{CONFIG_LIST}=             @{CONFIG01}    @{CONFIG02}    @{CONFIG03}    @{CONFIG04}
+        ```
+
+    + Run a simple test to verify the config is working correctly - for example
+      custom boot menu key:
+
+        ```robot
+        robot -v snipeit:no -L TRACE -v rte_ip:192.168.10.174 -v device_ip:0.0.0.0 -v config:pcengines-apu4 dasharo-compatibility/custom-boot-menu-key.robot
+        ```
+
+        If everything went right, the output may look something like this
+
+        ```bash
+        ==============================================================================
+        Custom-Boot-Menu-Key
+        ==============================================================================
+        CBK001.001 Custom boot menu key :: Check whether the DUT is config... | PASS |
+        ------------------------------------------------------------------------------
+        CBK002.001 Custom setup menu key :: Check whether the DUT is confi... | PASS |
+        ------------------------------------------------------------------------------
+        Custom-Boot-Menu-Key                                                  | PASS |
+        2 tests, 2 passed, 0 failed
+        ==============================================================================
+        Output:  /home/michal/Development/Dasharo/osfv/output.xml
+        Log:     /home/michal/Development/Dasharo/osfv/log.html
+        Report:  /home/michal/Development/Dasharo/osfv/report.html
+        ```
+
+## Adding new variant of an existing platform
+
+Some boards come in multiple variants, where the majority of properties and
+features can be shared. For these cases, we have shared "base" configs in
+`platform-configs/include/`. This way we don't need to copy-paste entire config
+files, making maintenance easier. In this example we'll be adding a new PC
+Engines apu variant, using an existing pcengines base config:
+
+- Create a config file in `platform-configs` for your platform
+- Add the following to your platform config
+
+```robot
+*** Settings ***
+Resource    include/pcengines.robot
+```
+
+- Add variant-specific settings for your platform - in this case, only the
+  SMBIOS product name field:
+
+```robot
+*** Variables ***
+${DMIDECODE_PRODUCT_NAME}=      apu4
+```
+
+- Proceed with adding the platform to `variables.robot` as per
+[Adding a brand new platform](#adding-a-brand-new-platform).

config-parser.md

@@ -0,0 +1,89 @@
+<!--
+SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Config parser
+
+## Intro
+
+One significant bit of work when adding a new platform, or updating the
+platform config for a new release, is listing all the features and looking up
+what test flags they correspond to. The `config_parser.py` script seeks to
+automate at least part of that process.
+
+It takes the config file used to build the binary, where most if not all
+features of the release should be encoded, and outputs a platform config file.
+
+## Usage
+
+The script should be used in the following manner:
+
+```sh
+ ./scripts/config_parser.py config.input output.robot
+```
+
+Where `config.input` is the coreboot config file, and `output.robot` is the
+resulting OSFV platform config file. Both the full `.config` and a defconfig
+can be used, although the `.config` will probably yield more matches.
+
+## Extendability, contribution
+
+The list of known option:flag pairs and value remapping rules are set in the
+`scripts/lib/mappings.json` file.
+
+By default, the script maps `y` to `${TRUE}` and `n` to `${FALSE}` or simply
+copies over the value from the option to the flag. This behavior can be
+overridden by editing the aforementioned json file. The format is rather easy
+to figure out - it works as follows:
+
+```json
+{
+  "options": {
+    "CONFIG_OPTION_1": "TARGET_TEST_FLAG_1",
+    "CONFIG_OPTION_2": "TARGET_TEST_FLAG_2"
+  },
+  "values": {
+    "TARGET_TEST_FLAG_1": {
+      "CONFIG_OPTION_1_VALUE_1": "TARGET_TEST_FLAG_1_VALUE_1",
+      "CONFIG_OPTION_1_VALUE_2": "TARGET_TEST_FLAG_1_VALUE_2"
+    },
+    "TARGET_TEST_FLAG_2": {
+      "CONFIG_OPTION_2_VALUE_1": "TARGET_TEST_FLAG_2_VALUE_1",
+      "CONFIG_OPTION_2_VALUE_2": "TARGET_TEST_FLAG_2_VALUE_2"
+    }
+  }
+}
+```
+
+So, for example:
+
+```json
+{
+  "options": {
+    "CONFIG_MAINBOARD_POWER_FAILURE_STATE": "DEFAULT_POWER_STATE_AFTER_FAIL",
+  },
+  "values": {
+    "DEFAULT_POWER_STATE_AFTER_FAIL": {
+      "0": "Powered Off",
+      "1": "Powered On",
+      "2": "The state at the moment of power failure"
+    }
+  }
+}
+```
+
+will remap `CONFIG_MAINBOARD_POWER_FAILURE_STATE=0` to
+`${DEFAULT_POWER_STATE_AFTER_FAIL}=    Powered Off`, and so on.
+
+If you see a good option:flag pair that can be added, a contribution will be
+very much welcome. You have a chance of speeding up the process for yourself
+and for others in the future.
+
+## Future improvements
+
+The entire `.config` is stored in each of our release binaries. Further
+automation could involve extracting the config from the binary with
+`cbfstool coreboot.rom extract -n config -f dotconfig` and updating the config
+with the parser automatically, e.g., as part of the regression testing process.

contributing.md

@@ -0,0 +1,96 @@
+<!--
+SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Contributing
+
+## Code
+
+* Install pre-commit hooks after cloning repository:
+
+```bash
+pre-commit install
+```
+
+## Issues
+
+If you are certain that the issue is related to this repository, create issue
+directly
+[here](https://github.com/Dasharo/open-source-firmware-validation/issues/new/choose).
+Otherwise, create an issue in
+[dasharo-issues repisotory](https://github.com/Dasharo/dasharo-issues/issues/new/choose).
+
+## Guidelines
+
+A list of guidelines we shall follow during transition to improve the quality
+of this repository. We start with getting rid of duplicated keywords, reducing
+the size of `keywords.robot` file, and improving their overall quality.
+
+There are other areas of interest that we will look into in the next steps
+and add as guidelines:
+* variables (use Python/YAML, not robot syntax),
+* platform-configs (get rid of duplication, and unused data),
+* separate test for different OS into different suites,
+* prepare the OS for running test suite via some dedicated tools (e.g. Ansible),
+  rather than implementing keywords for that from scratch,
+* reduce the number of unnecessary power events, so tests can finish quicker,
+* improve overall code quality by enabling back more
+  [robocop checks we cannot pass right now](https://github.com/Dasharo/open-source-firmware-validation/blob/main/robocop.toml),
+* To Be Continued.
+
+### Pre-commit and CI checks
+
+1. Make sure to use `pre-commit` locally. All pre-commit and other CI checks
+   must pass of course, prior requesting for review. Please check the status of
+   checks in your PR. If the failure is questionable, provide your arguments
+   for that, rather than silently ignoring this fact.
+
+### Code style
+
+1. It is automatically handled by
+  [robotidy](https://robotidy.readthedocs.io/en/stable/). The current rules
+  can be found
+  [here](https://github.com/Dasharo/open-source-firmware-validation/blob/main/.robotidy).
+
+### Keywords
+
+1. No new keywords in `keywords.robot` will be accepted
+* new keywords must be placed in a logically divided modules, under `lib/`
+      directory
+    - see
+        [openbmc-test-automation](https://github.com/openbmc/openbmc-test-automation/tree/master/lib)
+      as a reference
+* if you need to modify something in `keywords.robot`, you should create a new
+      module under `lib/`
+* if you add new keyword module, you should review the `keywords.module` and
+      move related keywords there as well, if suitable
+1. If keyword from keywords.robot can be reused or improved, do that instead
+   of creating a new one
+   - keyword duplication will not be accepted,
+   - you will be asked to use/improve existing keywords instead.
+1. You are encouraged to use Python for more sophisticaed or complex keywords
+   (e.g. more convoluted data parsing and processing). We are not forced to use
+   RF for all keywords. Especially when it is simply easier to use Python.
+1. For reading from terminal (no matter if it is Telnet, or SSH),
+   following keywords must be used:
+   - `Read From Terminal Until Prompt`
+   - `Read From Terminal Until`
+   - `Read From Terminal`
+   Usage of other keywords is prohibited. Whenever you modify a test/keyword,
+   you should rework it to use one of the above.
+1. For writing into terminal, following keywords must be used:
+   - `Execute Command In Terminal`
+   - `Write Into Terminal`
+   - `Write Bare Into Terminal`
+   Usage of other keywords is prohibited. Whenever you modify a test/keyword,
+   you should rework it to use one of the above.
+   You should use `Execute Command In Terminal` unless you have a very good
+   reason not to. Thanks to that, your keyword will not leave floating output
+   in buffer to be received by another keywords, not expecting that.
+
+### Documentation
+
+* Each new (or modified) file, test, keyword, must have a `[Documentation]`
+  section.

dts-tests.md

@@ -0,0 +1,50 @@
+<!--
+SPDX-FileCopyrightText: 2024 3mdeb <[email protected]>
+
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# DTS tests
+
+This document describes current DTS tests coverage. Tests planned:
+
+* E2E tests;
+* Unit tests.
+
+## E2E tests
+
+E2E - End to End tests, tests that verify how DTS goes from start to end for
+every user workflow (e.g. installation, update, etc.) for every platform.
+
+Location in OSFV: `dts/dts-e2e.robot`.
+
+These tests include modifications for DTS platform emulation so the tests
+could be launched on Qemu. Then every test case choose the workflow by
+choosing DTS menu option, and provides credentials (if necessary). So, the
+start conditions are: platform configuration and workflow selection.
+
+Then every test case goes through the chosen workflow and checks for expected
+behavior. If everything goes as expected - test case passes, otherwise - it
+fails.
+
+Control variables:
+
+* `dts_ipxe_link`: useful if you are testing DTS which is not released yet. Just
+  put here a link to your script which will load your DTS. By default DTS is
+  being booted from `dl.3mdeb.com`;
+* `dpp_logs_key`, `dpp_download_key`, `dpp_password`: for DPP credentials, if
+  tests need them.
+
+Launching example:
+
+```bash
+robot -b command_log.txt -v snipeit:no -L TRACE -v config:qemu -v rte_ip:127.0.0.1 -v boot_dts_from_ipxe_shell:True -v dts_ipxe_link:http://192.168.0.102:8080/ipxe -v dpp_logs_key:'LOGS_KEY' -v dpp_download_key:'DOWNLOAD_KEY' -v dpp_password:'PASSWORD' -t "E2E006.002*" dts/dts-e2e.robot
+```
+
+> Note: replace `LOGS_KEY`, `DOWNLOAD_KEY` and `PASSWORD` with appropriate
+> credentials if required. `http://192.168.0.102:8080/ipxe` with your DTS iPXE
+> script link.
+
+## Unit tests
+
+These tests have not been implemented yet.