v0.11.9 Testing checklist

[jorgeepc]

Testing Tracetest

Latest release: https://github.com/kubeshop/tracetest/releases/tag/v0.11.9-rc.3

This is the entire checklist on what we should do to assert that Tracetest is working fine on each version release. On each version release, we can copy the contents of this template checklist and open a Github Discussion to start each test.

On Tracetest, we work in two ways to test the entire system and guarantee that everything is working fine. One is through automatic tests, where we run unit, integration and end-to-end tests on Tracetest CLI, Web UI and Server.

Another source of tests is the manual tests that we execute on each release, simulating users, and checking if everything is ok on these tests.

Manual Tests

On our manual tests, we aim to do some sanity checks to see if the main features are working as expected. Usually, we run it on each Tracetest release.

Tests to validate on Release Candidate

• Test server installation via CLI

  • Docker Compose and no demo API (@mathnogueira on Linux)
  • Docker Compose and demo API (@mathnogueira on Linux)
  • Kubernetes and no demo API (@schoren)
  • Kubernetes and demo API (@schoren)
  • Verify that Tracetest is not sending analytics if you have the TRACETEST_DEV env for docker install (@mathnogueira)
  • Verify that Tracetest is not sending analytics if you have the TRACETEST_DEV env for k8s install (@schoren)

• Test DataStore provisioning and basic test

  • Open WebUI and go to /settings page. The provisioned Data Store should be selected
  • Create a test on WebUI that calls a demo API (like Pokeshop or Open Telemetry Store). This test should fetch traces correctly and run without errors

• Test Tracetest examples (We haven't changed code related to this, prob we can skip this step)

  • Jaeger example
  • Tempo example
  • New Relic example
  • Lightstep example
  • OpenSearch example
  • Elastic APM example
    • Tracetest is working, however, we found a bug due to an update on Elastic infra: ElasticAPM examples are not working. #2591
  • SignalFX example
  • Datadog example
  • Amazon X-Ray example
  • Honeycomb

• Test specific features added/changed on this release:

  • Validate fix for apply environment --file. We now should be able to create a new env with id set on file, and also update it. (@schoren)
  • Validate new CLI transaction resource: list transaction, apply transaction, get transaction, delete transaction (@schoren)
  • Transactions CLI command: tracetest apply transaction -f [new-transaction-file]
    • has error with relative paths. fix: fix: correctly handle transaction relative paths #2582
  • Transactions CLI command: tracetest apply transaction -f [existing-transaction-file] (@schoren)
  • Transactions CLI command: tracetest delete transaction --id [existing-id] (@schoren)
  • Transactions CLI command: tracetest delete transaction --id [non-existing-id] (@schoren)
  • Transactions CLI command: tracetest export transaction --id current --file [transaction-file]
    • Not working: cli export inconsisntecies #2586
  • Transactions CLI command: tracetest get transaction --id [non-existing-id] (@schoren)
  • Transactions CLI command: tracetest get transaction --id [existing-id] --output pretty (@schoren)
  • Transactions CLI command: tracetest get transaction --id [existing-id] --output json (@schoren)
  • Transactions CLI command: tracetest get transaction --id [existing-id] --output yaml (@schoren)
  • Transactions CLI command: tracetest list transaction --output pretty (@schoren)
  • Transactions CLI command: tracetest list transaction --output json (@schoren)
  • Transactions CLI command: tracetest list transaction --output yaml (@schoren)
  • Transactions CLI command: tracetest list transaction --skip 1 --take 2 (@schoren)
  • Transactions CLI command: tracetest list transaction --sortBy name --sortDirection desc (@schoren)
  • Validate CLI test run command using a transaction definition (@schoren)
  • Validate webUI transaction CRUD (@schoren)
  • Validate webUI transaction run (@schoren)
  • Validate output from CLI list, apply, get, delete for resource datastore. Related to output normalization (@schoren, @danielbdias )
    • panic on server fix: improve resourcemanager handling of augmented header #2584
    • error when listing in no-tracing mode Fixing corner case of listing datastores when on no-tracing mode #2585
  • Validate output from CLI list, apply, get, delete for resource environment. Related to output normalization (@schoren)
    • panic on server fix: improve resourcemanager handling of augmented header #2584
  • Validate output from CLI list, apply, get, delete for resource transaction. Related to output normalization (@schoren)
  • Validate CLI dashboard command (@schoren)
  • Validate CLI version command (@schoren)
  • Validate context flag on server install command using --run-environment kubernetes --mode with-demo --kubernetes-context xxxx (@schoren)
  • Validate context flag on server install command using --run-environment kubernetes --mode just-tracetest --kubernetes-context xxxx
  • Validate context flag on server install command using --run-environment docker --mode with-demo
  • Validate context flag on server install command using --run-environment docker --mode just-tracetest
  • Validate if it's possible to have attributes with dashes in it's name

Tests to validate on Final Release

• Pipeline

  • Check if our release pipeline on Release Tracetest workflow on Github Actions worked correctly

• Test CLI updates

  • MacOS via homebrew
  • MacOS via curl script
  • Windows via chocolatey
  • Windows via manual download

• Docs

  • Double check Detailed installation docs and see if everything is documented correctly

• Test specific features added/changed on this release:

  • Add list of features here

Automatic Tests

Today Tracetest has 3 main components: a WebUI, a CLI and a Server.

Web UI

• Unit tests: Run by executing npm test on ./web folder
• End-to-end tests: Run using cypress against a temporary Tracetest created on Kubernetes.

CLI

• Unit tests: Run by executing make test on ./cli folder

Server

• Unit tests: Run by executing make test on ./server folder
• Integration tests: Run along with some unit tests running make test on ./server folder, but also done in a matrix test on Github actions, by executing the test-examples action.
• End-to-end tests: Run using Tracetest to test itself. We deploy two instances of Tracetest and use one to test API calls to another on Github actions, by executing the trace-testing action.

[danielbdias]

list datastore with empty datastores return error:

[schoren]

panic on server when getting non-augmentable entities as json

[schoren]

bug with exporting transactions: #2586