Add OCR test run book

smartcontractkit · Sep 30, 2024 · ae0e26b · ae0e26b
1 parent 95bffd2
commit ae0e26b
Show file tree

Hide file tree

Showing 5 changed files with 330 additions and 42 deletions.
diff --git a/.github/E2E_TESTS_ON_GITHUB_CI.md b/.github/E2E_TESTS_ON_GITHUB_CI.md
@@ -1,6 +1,6 @@
 # E2E Tests on GitHub CI
 
-E2E tests are executed on GitHub CI using the [E2E Tests Reusable Workflow](#about-the-reusable-workflow) or dedicated workflows.
+E2E tests are executed on GitHub CI using the [E2E Tests Reusable Workflow](https://github.com/smartcontractkit/.github/blob/main/.github/workflows/README.md) or dedicated workflows.
 
 ## Automatic workflows
 
@@ -49,10 +49,9 @@ To learn more about test configs see [CTF Test Config](https://github.com/smartc
 
 ## Test Secrets
 
+> [!IMPORTANT]
+> "On demand" workflows are using default test secrets (e.g. a URL to a private Docker registry). Hence, it is recommended to upload your secrets to GitHub to avoid test fails caused by default values.
+
 For security reasons, test secrets and sensitive information are not stored directly within the test config TOML files. Instead, these secrets are securely injected into tests using environment variables. For a detailed explanation on managing test secrets, refer to our [Test Secrets documentation](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets).
 
 If you need to run a GitHub workflow using custom secrets, please refer to the [guide on running GitHub workflows with your test secrets](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#run-github-workflow-with-your-test-secrets).
-
-## About the E2E Test Reusable Workflow
-
-For information on the E2E Test Reusable Workflow, visit the documentation in the [smartcontractkit/.github repository](https://github.com/smartcontractkit/.github/blob/main/.github/workflows/README.md).
diff --git a/integration-tests/Makefile b/integration-tests/Makefile
@@ -102,27 +102,35 @@ test_smoke_product: ## Run smoke tests for specific product ex: make test_smoke_
 
 # Chaos
 .PHONY: test_chaos_pods_raw
-test_chaos_pods_raw:
+test_chaos_pods_raw: ## Run all chaos pod tests
 	go test -timeout 2h -v -count=1 $(args) -p 2 -run 'Test/.*pod-chaos' ./chaos
 
 .PHONY: test_chaos_network_raw
-test_chaos_network_raw:
+test_chaos_network_raw: ## Run all chaos network tests
 	go test -timeout 2h -v -count=1 $(args) -p 2 -run 'Test/.*network-chaos' ./chaos
 
 .PHONY: test_chaos_pods
-test_chaos_pods: install_gotestloghelper ## Run all chaos pod tests
+test_chaos_pods: install_gotestloghelper ## Run all chaos pod tests with decorated output
 	TEST_LOG_LEVEL="disabled" \
 	go test -timeout 2h -count=1 -json $(args) -run 'Test/.*pod-chaos' ./chaos 2>&1 | tee ./gotest.log | gotestloghelper -json -hidepassingtests -tlogprefix -color -singlepackage
 
 .PHONY: test_chaos_network
-test_chaos_network: install_gotestloghelper ## Run all chaos network tests
+test_chaos_network: install_gotestloghelper ## Run all chaos network tests with decorated output
 	TEST_LOG_LEVEL="disabled" \
 	go test -timeout 2h -count=1 -json $(args) -run 'Test/.*network-chaos' ./chaos 2>&1 | tee ./gotest.log | gotestloghelper -json -hidepassingtests -tlogprefix -color -singlepackage
 
 .PHONY: test_chaos_verbose
 test_chaos_verbose: ## Run all smoke tests with verbose logging
 	go test -timeout 24h -count=1 -v $(args) ./chaos
 
+.PHONY: test_chaos_ocr
+test_chaos_ocr: ## Run only OCR chaos tests
+	go test -timeout 2h -v -count=1 $(args) -p 2 -run 'TestOCRChaos' ./chaos
+
+.PHONY: test_chaos_automation
+test_chaos_automation: ## Run only Automation chaos tests
+	go test -timeout 2h -v -count=1 $(args) -p 2 -run 'TestAutomationChaos' ./chaos
+
 # Migrations
 .PHONY: test_node_migrations
 test_node_migrations: install_gotestloghelper ## Run all node migration tests.

diff --git a/integration-tests/README.md b/integration-tests/README.md
@@ -6,11 +6,12 @@
     - [Pre-requisites](#pre-requisites)
       - [Test and node configuration](#test-and-node-configuration)
     - [Run Tests](#run-tests)
-      - [Locally](#locally)
+      - [Locally (in Docker)](#locally-in-docker)
         - [All tests in a suite](#all-tests-in-a-suite)
         - [A single test](#a-single-test)
-      - [In CI](#in-ci)
       - [In Kubernetes](#in-kubernetes)
+        - [From local machine](#from-local-machine)
+      - [CI/GitHub Actions](#cigithub-actions)
 
 ## Summary
 
@@ -25,15 +26,19 @@ This directory represent a place for different types of integration and system l
 ### Pre-requisites
 
 1. [Installed Go](https://go.dev/)
-2. For local testing, [Installed Docker](https://www.docker.com/). Consider [increasing resources limits needed by Docker](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) as most tests require building several containers for a Decentralized Oracle Network (e.g. OCR requires 6 nodes and DBs, a simulated blockchain, and a mock server).
+2. For local testing, [Installed Docker](https://www.docker.com/). Consider [increasing resources limits needed by Docker](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) as most tests require building several containers for a Decentralized Oracle Network (e.g. OCR requires 6 nodes, 6 DBs, and a mock server).
 3. For remote testing, access to Kubernetes cluster/AWS Docker registry (if you are pulling images from private links).
 4. Docker image. If there is no image to pull from a registry, you may run tests against a custom build. Run the following command to build the image:
 
-```bash
-make build_docker_image image=<your-image-name> tag=<your-tag>
-```
+    ```bash
+    make build_docker_image image=<your-image-name> tag=<your-tag>
+    ```
 
-Example: `make build_docker_image image=chainlink tag=test-tag`
+    Example: `make build_docker_image image=chainlink tag=test-tag`
+
+5. RPC node/s (for testnets/mainnets).
+6. EOA's (wallet) Private Key (see [How to export an account's private key](https://support.metamask.io/ru/managing-my-wallet/secret-recovery-phrase-and-private-keys/how-to-export-an-accounts-private-key/))
+7. Sufficient amount of native token and LINK on EOA per a target chain.
 
 #### Test and node configuration
 
@@ -43,55 +48,117 @@ Example: `make build_docker_image image=chainlink tag=test-tag`
       1. `cd integration-tests`
       2. `source .env`
 
-2. Setup test secrets. See "how-to" details in the [CTF config README](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets).
+2. Setup test secrets. See "how-to" details in the [CTF config README](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets). If you want to run tests in CI, you will have to push test secrets to GitHub (see [Run GitHub Workflow with your test secrets](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#run-github-workflow-with-your-test-secrets)).
 
 3. Provide test and node configuration (for more details refer to [testconfig README](./testconfig/README.md) and `example.toml` files):
    1. **Defaults** for all products are defined in `./testconfig/<product>/<product>.toml` files.
    2. To **override default values**, create a `./testconfig/overrides.toml` file (yes, in the root of `testconfig`, not a product directory) specifying the values to override by your test (see some examples in [./testconfig/ocr2/overrides](./testconfig/ocr2/overrides)).
     > [!IMPORTANT]
     > **Image version and node configs**
-    > 1. Do not forget to set `[ChainlinkImage].version` to test against the necessary remotely accessible version or [custom build](#optional-build-docker-image).
+    > 1. Pay attention to the `[ChainlinkImage].version` to test against the necessary remotely accessible version or [custom build](#pre-requisites).
     > 2. When running OCR-related tests, pay attention to which version of OCR you enable/override in your `overrides.toml`.
-    > 3. Pay attention to not committing any sensitive data.
+    > 3. Do not commit any sensitive data.
 
 4. [Optional] Configure Seth (or use defaults), an evm client used by tests. Detailed instructions on how to configure it can be found in the [Seth README](./README_SETH.md) and [Seth repository](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/seth).
 
+   > [!IMPORTANT]
+   > **Simulated mode (no test secrets needed)**
+   > Tests may run in a simulated mode, on a simulated chain (1337). In the `overrides.toml` file, set the following:
+   > 1. `[Network].selected_networks=["simulated"]`
+   > 2. `[[Seth.networks]].name = "Default"`
+
 ### Run Tests
 
-#### Locally
+#### Locally (in Docker)
 
 > [!NOTE]
-> **Resources utilization**
+> **Resources utilization by Docker**
 > It's recommended to run only one test at a time (run tests sequentially) on a local machine as it needs a lot of docker containers and can peg your resources otherwise. You will see docker containers spin up on your machine for each component of the test where you can inspect logs.
 
 ##### All tests in a suite
 
-1. Run CLI command: `go test ./smoke/<product>_test.go`
-   Example: `go test ./smoke/ocr_test.go`
-2. Logs of each Chainlink container will dump into the `smoke/logs/`.
-3. To enable debugging of HTTP and RPC clients set the following env vars:
+1. Run CLI command(with `override.toml`):
 
-```bash
-export SETH_LOG_LEVEL=debug
-export RESTY_DEBUG=true
-```
+   ```bash
+   BASE64_CONFIG_OVERRIDE=$(cat ./testconfig/overrides.toml | base64) go test -v -p 1 ./smoke/<product>_test.go
+   ```
+
+   Example:
+
+   ```bash
+   BASE64_CONFIG_OVERRIDE=$(cat ./testconfig/overrides.toml | base64) go test -v -p 1 ./smoke/ocr_test.go
+   ```
+
+   > [!WARNING]
+   > **Parallelized tests and nonce issues**
+   > Most tests are paralelized by default. To avoid nonce-related issues, it is recommended to run tests with disabled parallelization, e.g. with `-p 1`.
+
+2. Alternatively, you may use `make` commands (see more in [Makefile .PHONY lines](./Makefile)) for running suites of tests.
+    Example:
+
+    ```bash
+    make test_smoke_product product="ocr" ./scripts/run_product_tests
+    ```
+
+3. Logs of each Chainlink container will dump into the `smoke/logs/`.
+4. To enable debugging of HTTP and RPC clients set the following env vars:
+
+    ```bash
+    export SETH_LOG_LEVEL=debug
+    export RESTY_DEBUG=true
+    ```
 
 ##### A single test
 
-1. Run CLI command: `go test ./smoke/<product>_test.go -run <TestNameToRun>`
-   Example: `go test ./smoke/ocr_test.go -run TestOCRBasic`
+Run CLI command (with `override.toml`):
+
+```bash
+BASE64_CONFIG_OVERRIDE=$(cat ./testconfig/overrides.toml | base64) go test -v -timeout 15m -run <"TestNameToRun"> ./<directory_name_with_tests>
+```
 
-#### In CI
+Example:
 
-1. Refer [Tests Run Books](./run-books/) for more details.
-2. Logs in CI uploaded as GitHub artifacts.
+```bash
+BASE64_CONFIG_OVERRIDE=$(cat ./testconfig/overrides.toml | base64) go test -v -timeout 15m -run "TestOCRv2Basic" ./smoke
+```
 
 #### In Kubernetes
 
 Such tests as Soak, Performance, Benchmark, and Chaos Tests remain bound to a Kubernetes run environment.
 
-1. Ensure all necessary configuration is provided (see [Test Configuration](#test-configuration))
-2. You are logged in to your Kubernetes cluster (with `aws sso login`)
-3. Run CLI command: `make test_<your_test>` (see commands in [Makefile .PHONY lines](./Makefile) for more details)
+1. Refer [Tests Run Books](./run-books/) to get more details on how to run specific per-product tests.
+2. Logs in CI are uploaded as GitHub artifacts.
+
+##### From local machine
+
+1. Ensure all necessary configurations are provided (see [Test and node configuration](#test-and-node-configuration))
+2. [Optional] Encode `override.toml`, if any, with the following command:
+
+    ```bash
+    cat testconfig/overrides.toml | base64
+    ```
+
+3. Copy and paste the output of the encoded `override.toml` into `BASE64_CONFIG_OVERRIDE` var in `.env` file to be later used by Kubernetes.
+   Example:
+
+   ```bash
+   export BASE64_CONFIG_OVERRIDE='Base_64_encoded_content_of_override.toml'
+   ```
+
+   Ensure to **update you environment** with the following command:
+
+   ```bash
+   source .env
+   ```
+
+4. Log in to your Kubernetes cluster (with `aws sso login`)
+5. Run test with the following CLI command: `make test_<your_test>` (see make-commands in [Makefile .PHONY lines](./Makefile)).
    Example: `make test_soak_ocr`, `make test_soak_ocr2`, `test_node_migrations`, etc.
-4. Navigate to Grafana dashboards to see test and node logs, and results.
+6. Use Kubernetes namespace printed out in logs to monitor and analyze test runs.
+7. Navigate to Grafana dashboards to for test results and logs.
+
+#### CI/GitHub Actions
+
+1. Ensure all necessary configurations are provided (see [Test and node configuration](#test-and-node-configuration)).
+2. Follow instructions provided in [E2E Tests on GitHub CI](../.github/E2E_TESTS_ON_GITHUB_CI.md).
+3. Refer [Tests Run Books](./run-books/) to get more details on how to run specific per-product tests.