Skip to content

Commit

Permalink
Update integration-tests README
Browse files Browse the repository at this point in the history
  • Loading branch information
@chudilka1
chudilka1 committed Sep 26, 2024
1 parent 2440917 commit 95bffd2
Show file tree
Hide file tree
Showing 3 changed files with 133 additions and 107 deletions.
168 changes: 64 additions & 104 deletions integration-tests/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,137 +1,97 @@
# Integration Tests

Here lives the integration tests for chainlink, utilizing our [chainlink-testing-framework](https://github.com/smartcontractkit/chainlink-testing-framework).
- [Integration Tests](#integration-tests)
- [Summary](#summary)
- [Guidelines](#guidelines)
- [Pre-requisites](#pre-requisites)
- [Test and node configuration](#test-and-node-configuration)
- [Run Tests](#run-tests)
- [Locally](#locally)
- [All tests in a suite](#all-tests-in-a-suite)
- [A single test](#a-single-test)
- [In CI](#in-ci)
- [In Kubernetes](#in-kubernetes)

## NOTE: Move to Testcontainers
## Summary

If you have previously run these smoke tests using GitHub Actions or some sort of Kubernetes setup, that method is no longer necessary. We have moved the majority of our tests to utilize plain Docker containers (with the help of [Testcontainers](https://golang.testcontainers.org/)). This should make tests faster, more stable, and enable you to run them on your local machine without much hassle.
This directory represent a place for different types of integration and system level tests. It utilizes [Chainlink Testing Framework (CTF)](https://github.com/smartcontractkit/chainlink-testing-framework).

## Requirements
> [!TIP]
> **Testcontainers (Dockerized tests)**
> If you want to have faster, locally running, more stable tests, utilize plain Docker containers (with the help of [Testcontainers](https://golang.testcontainers.org/)) instead of using GitHub Actions or Kubernetes.
1. [Go](https://go.dev/)
2. [Docker](https://www.docker.com/)
3. You'll probably want to [increase the resources available to Docker](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) as most tests require quite a few containers (e.g. OCR requires 6 Chainlink nodes, 6 databases, a simulated blockchain, and a mock server).
## Guidelines

## Configure
### Pre-requisites

We have finished the first pass at moving all test configuration from env vars to TOML files. All product-related configuration is already in TOML files, but env vars are still used to control the log level, Slack notifications, and Kubernetes-related settings. See the [example.env](./example.env) file for how to set these environment variables.

We have defined some sensible defaults for all products, you can find them in `./testconfig/<product>/<product>.toml` files. Each product folder contains an `example.toml` file that describes all options. If you wish to override these values, you can do so by creating a `./testconfig/overrides.toml`. A detailed description of TOML configuration can be found in the [testconfig README](./testconfig/README.md), but if you want to run some tests using default values all you need to do is provide the Chainlink image and version you want to run tests on:
```toml
# ./testconfig/overrides.toml

[ChainlinkImage]
image = "your image name"
version = "your tag"
```

The `./testconfig/overrides.toml` file **should never be committed** and has been added to the [.gitignore](../.gitignore) file as it can often contain secrets like private keys and RPC URLs.

For more information on how to configure the tests, see the [testconfig README](./testconfig/README.md).

## Build

If you'd like to run the tests on a local build of Chainlink, you can point to your own docker image, or build a fresh one with `make`.

`make build_docker_image image=<image-name> tag=<tag>`

e.g.

`make build_docker_image image=chainlink tag=test-tag`

## Run

Ensure you have created a `./testconfig/overrides.toml` file with your desired Chainlink image and version.

`go test ./smoke/<product>_test.go`

Most test files have a couple of tests, it's recommended to look into the file and focus on a specific one if possible. 90% of the time this will probably be the `Basic` test. See [ocr_test.go](./smoke/ocr_test.go) for example, which contains the `TestOCRBasic` test.

`go test ./smoke/ocr_test.go -run TestOCRBasic`

It's generally recommended to run only one test at a time on a local machine as it needs a lot of docker containers and can peg your resources otherwise. You will see docker containers spin up on your machine for each component of the test where you can inspect logs.

### Configure Seth

Our new evm client is Seth. Detailed instructions on how to configure it can be found in the [Seth README](./README_SETH.md) in this repo as well as in [Seth repository](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/seth).

## Analyze

You can see the results of each test in the terminal with normal `go test` output. If a test fails, logs of each Chainlink container will dump into the `smoke/logs/` folder for later analysis. You can also see these logs in CI uploaded as GitHub artifacts.

## Running Soak, Performance, Benchmark, and Chaos Tests

These tests remain bound to a Kubernetes run environment, and require more complex setup and running instructions not documented here. We endeavor to make these easier to run and configure, but for the time being please seek a member of the QA/Test Tooling team if you want to run these.

### How to run reorg tests
Run soak/ocr_test.go with reorg below finality and `FinalityTagEnabled=false`
1. [Installed Go](https://go.dev/)
2. For local testing, [Installed Docker](https://www.docker.com/). Consider [increasing resources limits needed by Docker](https://stackoverflow.com/questions/44533319/how-to-assign-more-memory-to-docker-container) as most tests require building several containers for a Decentralized Oracle Network (e.g. OCR requires 6 nodes and DBs, a simulated blockchain, and a mock server).
3. For remote testing, access to Kubernetes cluster/AWS Docker registry (if you are pulling images from private links).
4. Docker image. If there is no image to pull from a registry, you may run tests against a custom build. Run the following command to build the image:

```bash
make test_soak_ocr_reorg_1
make build_docker_image image=<your-image-name> tag=<your-tag>
```

Run soak/ocr_test.go with reorg below finality and `FinalityTagEnabled=true`:
Example: `make build_docker_image image=chainlink tag=test-tag`

```bash
make test_soak_ocr_reorg_2
```
#### Test and node configuration

Run reorg/automation_reorg_test.go with reorg settings:
1. Setup `.env` file in the root of `integration-tests` directory. See [example.env](./example.env) for how to set test-runner log level (not a node's log level), Slack notifications, and Kubernetes-related settings.

1. Use Simulated Geth network and put GethReorgConfig in overrides.toml
1. Ensure to **update you environment** with the following commands:
1. `cd integration-tests`
2. `source .env`

```toml
[Network]
selected_networks=["simulated"]
[Network.GethReorgConfig]
enabled = true
depth = 10
delay_create = "3s"
```
2. Setup test secrets. See "how-to" details in the [CTF config README](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets).

2. Then run the test:
```bash
make test_reorg_automation
```
3. Provide test and node configuration (for more details refer to [testconfig README](./testconfig/README.md) and `example.toml` files):
1. **Defaults** for all products are defined in `./testconfig/<product>/<product>.toml` files.
2. To **override default values**, create a `./testconfig/overrides.toml` file (yes, in the root of `testconfig`, not a product directory) specifying the values to override by your test (see some examples in [./testconfig/ocr2/overrides](./testconfig/ocr2/overrides)).
> [!IMPORTANT]
> **Image version and node configs**
> 1. Do not forget to set `[ChainlinkImage].version` to test against the necessary remotely accessible version or [custom build](#optional-build-docker-image).
> 2. When running OCR-related tests, pay attention to which version of OCR you enable/override in your `overrides.toml`.
> 3. Pay attention to not committing any sensitive data.
Run reorg above finality docker test:
4. [Optional] Configure Seth (or use defaults), an evm client used by tests. Detailed instructions on how to configure it can be found in the [Seth README](./README_SETH.md) and [Seth repository](https://github.com/smartcontractkit/chainlink-testing-framework/tree/main/seth).

```bash
go test -v -run ^TestReorgAboveFinality_FinalityTagDisabled$ ./smoke
```
### Run Tests

### How to run gas simulation tests
#### Locally

Run soak/ocr_test.go with gas spike:
> [!NOTE]
> **Resources utilization**
> It's recommended to run only one test at a time (run tests sequentially) on a local machine as it needs a lot of docker containers and can peg your resources otherwise. You will see docker containers spin up on your machine for each component of the test where you can inspect logs.
```bash
make test_soak_ocr_gas_spike
```
##### All tests in a suite

Run soak/ocr_test.go with changing gas limit creating block congestion:
1. Run CLI command: `go test ./smoke/<product>_test.go`
Example: `go test ./smoke/ocr_test.go`
2. Logs of each Chainlink container will dump into the `smoke/logs/`.
3. To enable debugging of HTTP and RPC clients set the following env vars:

```bash
make test_soak_ocr_gas_limit_change
export SETH_LOG_LEVEL=debug
export RESTY_DEBUG=true
```

Note: you can update gas simulation params for the tests below in in testconfig/ocr.toml
##### A single test

### How to run tests with RPC node failure
1. Run CLI command: `go test ./smoke/<product>_test.go -run <TestNameToRun>`
Example: `go test ./smoke/ocr_test.go -run TestOCRBasic`

Run soak/ocr_test.go with RPC network chaos by bringing down network to RPC node for all Chainlink Nodes:
#### In CI

```bash
make test_soak_ocr_rpc_down_all_cl_nodes
```
1. Refer [Tests Run Books](./run-books/) for more details.
2. Logs in CI uploaded as GitHub artifacts.

Run soak/ocr_test.go with RPC network chaos by bringing down network to RPC node for 50 percent of Chainlink Nodes:
#### In Kubernetes

```bash
make test_soak_ocr_rpc_down_half_cl_nodes
```
Such tests as Soak, Performance, Benchmark, and Chaos Tests remain bound to a Kubernetes run environment.

### Debugging HTTP and RPC clients
```bash
export SETH_LOG_LEVEL=debug
export RESTY_DEBUG=true
```
1. Ensure all necessary configuration is provided (see [Test Configuration](#test-configuration))
2. You are logged in to your Kubernetes cluster (with `aws sso login`)
3. Run CLI command: `make test_<your_test>` (see commands in [Makefile .PHONY lines](./Makefile) for more details)
Example: `make test_soak_ocr`, `make test_soak_ocr2`, `test_node_migrations`, etc.
4. Navigate to Grafana dashboards to see test and node logs, and results.
65 changes: 65 additions & 0 deletions integration-tests/run-books/VRF.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# VRF Tests Run Book
* All test configs should be placed in the [integration-tests/testconfig](../testconfig) folder
* All test configs for running tests in live testnets should be under [integration-tests/testconfig/vrfv2plus/overrides](../testconfig/vrfv2plus/overrides) folder


## Functional Tests
### In CI - using On Demand Workflows
```bash
gh workflow run "on-demand-vrfv2plus-smoke-tests.yml" \
--ref develop \
-f=test_secrets_override_key=<your testsecrets id> \
-f test_config_override_path=<path to test toml config which should be in `integration-tests/testconfig/vrfv2plus/overrides` folder> \
-f test_suite="Selected Tests" \ # Optional, Options - "All Tests", "Selected Tests". Default is "All Tests". If "Selected Tests" is selected, then `test_list_regex` should be provided
-f test_list_regex="<regex for tests to run>" \ # Optional, default is "TestVRFv2Plus$/(Link_Billing|Native_Billing|Direct_Funding)|TestVRFV2PlusWithBHS" which are P0 tests
-f chainlink_version="<>" # Optional, default is image created from develop branch. Not needed if you run tests against existing environment
-f notify_user_id_on_failure=<your slack user id> # Optional, default is empty. If provided, will notify the user on slack if the tests fail
```

#### Examples:
Run P0 tests against existing environment (Staging) on Arbitrum Sepolia
```bash
gh workflow run "on-demand-vrfv2plus-smoke-tests.yml" \
--ref develop \
-f=test_secrets_override_key=<your testsecrets id> \
-f test_config_override_path=integration-tests/testconfig/vrfv2plus/overrides/staging/arbitrum_sepolia_staging_test_config.toml \
-f test_suite="Selected Tests"
```

Run all tests deploying all contracts, CL nodes with `2.15.0` version on Base Sepolia
```bash
gh workflow run "on-demand-vrfv2plus-smoke-tests.yml" \
--ref develop \
-f=test_secrets_override_key=<your testsecrets id> \
-f test_config_override_path=integration-tests/testconfig/vrfv2plus/overrides/new_env/base_sepolia_new_env_test_config.toml \
-f test_suite="All Tests" \
-f chainlink_version="2.15.0"
```

### Locally
```bash
cd integration-tests
TEST_LOG_LEVEL=debug \
BASE64_CONFIG_OVERRIDE=$(cat <full path to chain's test toml config> | base64) \
go test -v -timeout 15m -run "<regex for tests to run>" ./smoke
```
## Performance Tests
```bash
gh workflow run "on-demand-vrfv2plus-performance-test.yml" \
--ref develop \
-f=test_secrets_override_key=<your testsecrets id> \
-f test_config_override_path=<path to test toml config which should be in `integration-tests/testconfig/vrfv2plus/overrides` folder> \
-f performanceTestType=“Smoke” # Options - "Smoke", "Soak", "Stress", "Load".
-f test_list_regex="<regex for tests to run>" # Optional, default is "TestVRFV2PlusPerformance"
```
#### Examples:
Run SOAK tests against existing environment (Staging) on Base Sepolia
```bash
gh workflow run "on-demand-vrfv2plus-performance-test.yml" \
--ref develop \
-f=test_secrets_override_key=<your testsecrets id> \
-f test_config_override_path=integration-tests/testconfig/vrfv2plus/overrides/staging/base_sepolia_staging_test_config.toml \
-f performanceTestType=“Soak”
```
7 changes: 4 additions & 3 deletions integration-tests/smoke/ocr_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,9 +105,6 @@ func prepareORCv1SmokeTestEnv(t *testing.T, l zerolog.Logger, firstRoundResult i
nodeClients := env.ClCluster.NodeAPIs()
bootstrapNode, workerNodes := nodeClients[0], nodeClients[1:]

err = actions.FundChainlinkNodesFromRootAddress(l, sethClient, contracts.ChainlinkClientToChainlinkNodeWithKeysAndAddress(workerNodes), big.NewFloat(*config.Common.ChainlinkNodeFunding))
require.NoError(t, err, "Error funding Chainlink nodes")

t.Cleanup(func() {
// ignore error, we will see failures in the logs anyway
_ = actions.ReturnFundsFromNodes(l, sethClient, contracts.ChainlinkClientToChainlinkNodeWithKeysAndAddress(env.ClCluster.NodeAPIs()))
Expand All @@ -122,6 +119,10 @@ func prepareORCv1SmokeTestEnv(t *testing.T, l zerolog.Logger, firstRoundResult i
err = actions.CreateOCRJobsLocal(ocrInstances, bootstrapNode, workerNodes, 5, env.MockAdapter, big.NewInt(sethClient.ChainID))
require.NoError(t, err, "Error creating OCR jobs")

// there is no need to fund the nodes unless the OCR contract and job are configured
err = actions.FundChainlinkNodesFromRootAddress(l, sethClient, contracts.ChainlinkClientToChainlinkNodeWithKeysAndAddress(workerNodes), big.NewFloat(*config.Common.ChainlinkNodeFunding))
require.NoError(t, err, "Error funding Chainlink nodes")

err = actions.WatchNewOCRRound(l, sethClient, 1, contracts.V1OffChainAgrregatorToOffChainAggregatorWithRounds(ocrInstances), time.Duration(3*time.Minute))
require.NoError(t, err, "Error watching for new OCR round")

Expand Down

0 comments on commit 95bffd2

Please sign in to comment.