Skip to content

Commit

Permalink
Convert CI python test arguments to YAML format (#35988)
Browse files Browse the repository at this point in the history
* Convert CI python test arguments to YAML format

* Drop support for non-YAML CI arguments format

Fixes #35787

* Rename factoryreset to factory-reset
  • Loading branch information
arkq authored Oct 9, 2024
1 parent d4c2eaf commit b3aeb67
Show file tree
Hide file tree
Showing 169 changed files with 2,603 additions and 1,354 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/tests.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -595,7 +595,7 @@ jobs:
"
- name: Run Tests
run: |
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --app out/darwin-x64-all-clusters-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factoryreset --quiet --app-args "--discriminator 3840 --interface-id -1" --script-args "-t 3600 --disable-test ClusterObjectTests.TestTimedRequestTimeout"'
scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --app out/darwin-x64-all-clusters-no-ble-no-wifi-tsan-clang-test/chip-all-clusters-app --factory-reset --quiet --app-args "--discriminator 3840 --interface-id -1" --script-args "-t 3600 --disable-test ClusterObjectTests.TestTimedRequestTimeout"'
- name: Uploading core files
uses: actions/upload-artifact@v4
if: ${{ failure() && !env.ACT }}
Expand Down
10 changes: 5 additions & 5 deletions docs/guides/matter-repl.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,8 +244,8 @@ Some tests provides the option to exclude them. For example, you can use
It is recommended to use the test wrapper to run mobile-device-test.py, for
example, you can run:

```
./scripts/tests/run_python_test.py --app chip-all-clusters-app --factoryreset
```shell
./scripts/tests/run_python_test.py --app chip-all-clusters-app --factory-reset
```

It provides some extra options, for example:
Expand All @@ -254,7 +254,7 @@ It provides some extra options, for example:
optional arguments:
-h, --help show this help message and exit
--app APP Path to local application to use, omit to use external apps.
--factoryreset Remove app config and repl configs (/tmp/chip* and /tmp/repl*) before running the tests.
--factory-reset Remove app config and repl configs (/tmp/chip* and /tmp/repl*) before running the tests.
--app-args APP_ARGS The extra parameters passed to the device side app.
--script SCRIPT Path to the test script to use, omit to use the default test script (mobile-device-test.py).
--script-args SCRIPT_ARGS Arguments for the REPL test script
Expand All @@ -263,8 +263,8 @@ optional arguments:
You can pass your own flags for mobile-device-test.py by appending them to the
command line with two dashes, for example:

```
./scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-no-ble-no-wifi-tsan-clang/chip-all-clusters-app --factoryreset --script-args "-t 90 --disable-test ClusterObjectTests.TestTimedRequestTimeout"
```shell
./scripts/tests/run_python_test.py --app out/linux-x64-all-clusters-no-ble-no-wifi-tsan-clang/chip-all-clusters-app --factory-reset --script-args "-t 90 --disable-test ClusterObjectTests.TestTimedRequestTimeout"
```

will pass `-t 90 --disable-test ClusterObjectTests.TestTimedRequestTimeout` to
Expand Down
10 changes: 6 additions & 4 deletions docs/testing/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Python tests located in src/python_testing
# --passcode 20202021
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factoryreset: true
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===
Expand Down Expand Up @@ -627,7 +627,9 @@ for the CI). These arguments can be passed as sets of key/value pairs using the
`./scripts/tests/run_python_test.py` is a convenient script that starts an
example DUT on the host and includes factory reset support

`./scripts/tests/run_python_test.py --factoryreset --app <your_app> --app-args "whatever" --script <your_script> --script-args "whatever"`
```shell
./scripts/tests/run_python_test.py --factory-reset --app <your_app> --app-args "whatever" --script <your_script> --script-args "whatever"
```

# Running tests in CI

Expand Down Expand Up @@ -678,7 +680,7 @@ for that run, e.g.:
# app: ${TYPE_OF_APP}
# app-args: <app_arguments>
# script-args: <script_arguments>
# factoryreset: <true|false>
# factory-reset: <true|false>
# quiet: <true|false>
# === END CI TEST ARGUMENTS ===
```
Expand All @@ -692,7 +694,7 @@ for that run, e.g.:

- Example: `${TYPE_OF_APP}`

- `factoryreset`: Determines whether a factory reset should be performed
- `factory-reset`: Determines whether a factory reset should be performed
before the test.

- Example: `true`
Expand Down
4 changes: 2 additions & 2 deletions scripts/tests/TestTimeSyncTrustedTimeSourceRunner.py
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,9 @@ def run_test_section(self, app_args: str, script_args: str, factory_reset_all: b
cmd = self.get_base_run_python_cmd(self.run_python_test_path, self.app_path, app_args,
self.script_path, script_args)
if factory_reset_all:
cmd = cmd + ' --factoryreset'
cmd = cmd + ' --factory-reset'
if factory_reset_app:
cmd = cmd + ' --factoryreset-app-only'
cmd = cmd + ' --factory-reset-app-only'

logging.info(f'Running cmd {cmd}')

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@
# --disable-test ClusterObjectTests.TestTimedRequestTimeout
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factoryreset: true
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TCP_Tests.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
# --passcode 20202021
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factoryreset: true
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===
#
Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_ACE_1_2.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# factoryreset: true
# factory-reset: true
# quiet: true
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_ACE_1_3.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# factoryreset: true
# factory-reset: true
# quiet: true
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_ACE_1_4.py
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
# --string-arg PIXIT.ACE.APPATTRIBUTE:OnOff
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factoryreset: true
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_ACE_1_5.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# factoryreset: true
# factory-reset: true
# quiet: true
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_ACL_2_11.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
# test-runner-runs:
# run1:
# app: ${NETWORK_MANAGEMENT_APP}
# factoryreset: true
# factory-reset: true
# quiet: true
# app-args: >
# --discriminator 1234 --KVS kvs1
Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_ACL_2_2.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# factoryreset: true
# factory-reset: true
# quiet: true
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
Expand Down
2 changes: 1 addition & 1 deletion src/python_testing/TC_AccessChecker.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# factoryreset: true
# factory-reset: true
# quiet: true
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
Expand Down
19 changes: 13 additions & 6 deletions src/python_testing/TC_BOOLCFG_2_1.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,19 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import functools
Expand Down
19 changes: 13 additions & 6 deletions src/python_testing/TC_BOOLCFG_3_1.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,19 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import logging
Expand Down
19 changes: 13 additions & 6 deletions src/python_testing/TC_BOOLCFG_4_1.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,19 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: --discriminator 1234 --KVS kvs1 --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import logging
Expand Down
24 changes: 18 additions & 6 deletions src/python_testing/TC_BOOLCFG_4_2.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,24 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --enable-key 000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: >
# --discriminator 1234
# --KVS kvs1
# --enable-key 000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import logging
Expand Down
24 changes: 18 additions & 6 deletions src/python_testing/TC_BOOLCFG_4_3.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,24 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --enable-key 000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: >
# --discriminator 1234
# --KVS kvs1
# --enable-key 000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import logging
Expand Down
24 changes: 18 additions & 6 deletions src/python_testing/TC_BOOLCFG_4_4.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,24 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --enable-key 000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: >
# --discriminator 1234
# --KVS kvs1
# --enable-key 000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import logging
Expand Down
24 changes: 18 additions & 6 deletions src/python_testing/TC_BOOLCFG_5_1.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,24 @@
#

# === BEGIN CI TEST ARGUMENTS ===
# test-runner-runs: run1
# test-runner-run/run1/app: ${ALL_CLUSTERS_APP}
# test-runner-run/run1/factoryreset: True
# test-runner-run/run1/quiet: True
# test-runner-run/run1/app-args: --discriminator 1234 --KVS kvs1 --enable-key 000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_APP}.json
# test-runner-run/run1/script-args: --storage-path admin_storage.json --commissioning-method on-network --discriminator 1234 --passcode 20202021 --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f --trace-to json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# test-runner-runs:
# run1:
# app: ${ALL_CLUSTERS_APP}
# app-args: >
# --discriminator 1234
# --KVS kvs1
# --enable-key 000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_APP}.json
# script-args: >
# --storage-path admin_storage.json
# --commissioning-method on-network
# --discriminator 1234
# --passcode 20202021
# --hex-arg PIXIT.BOOLCFG.TEST_EVENT_TRIGGER_KEY:000102030405060708090a0b0c0d0e0f
# --trace-to json:${TRACE_TEST_JSON}.json
# --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto
# factory-reset: true
# quiet: true
# === END CI TEST ARGUMENTS ===

import logging
Expand Down
Loading

0 comments on commit b3aeb67

Please sign in to comment.