Loki testing

[TheoBrigitte]

Loki Tests

These tests should be part of the PR Checks:

• Validate Helm chart templating
• Deploy chart
  • Kind (single instance)

These tests should be part of the tinkereres ci:

• Deploy chart
  • AWS, Azure, CAPI (later)
• Integration test; Verify all component work together
  • multi-tenant-proxy + Loki setup
  • Using canary to generate traffic

Inspired from

• https://github.com/grafana/loki/blob/main/.github/workflows/helm-ci.yml
• https://github.com/grafana/loki/blob/main/production/helm/loki/src/helm-test/canary_test.go

[QuantumEnigmaa]

The chart templating test is working.
Concerning the test of the chart's deployment on a kind cluster though, the circleci job is always failing. The apiserver test is passing, but the following ones are failing.

I tried to debug it by relaunching the failed job with ssh, ssh into the job's container to then exec into the ats docker container. There I was able to take a look at the kind cluster and here is what I saw :

• All loki pods are deployed and running except for the loki-multi-tenant-proxy one
• loki-write and loki-gateway pods were in ready state
• loki-read and loki-backend pods were not ready and the read ones logged errors about not being able to reach the loki-backend-headless service. It appears that those pods are not able to see other pods from their rings. Maybe due to the lack of network policies.

In any case, the loki deployments and statefulsets were indeed present on the cluster so the first test at least should have succeeded.

[QuantumEnigmaa]

Concerning e2e tests we discovered that the apptest-framework doesn't support MC-only app.

We thus decided to go another way and build our own custom "e2e" tests framework :

• We will update the PR message to empower the user to perform the required tests before asking reviews for its PR.
• A dedicated directory will be created at the repo's root with a file describing the test procedure and possible scripts.
• Concerning Loki specifically, the testing procedure will consist in deploying the PR's branch version to a giantswarm testing MC with canary enabled, let it run a bit and check on the dedicated dashboard that eveything is working well.

[QuantumEnigmaa]

Manual test procedure and script added to the app's repo