Setup logging on CAPA installations

[TheoBrigitte]

Towards: #311

In order to expand our logging infrastructure coverage we should extend support to CAPA installations

Tasks

Beta Give feedback

1. Support Object Storage on CAPA #2872
   component/crossplane epic/logging provider/cluster-api-aws topic/capa-ga topic/observability +5
   
2. Deploy Loki on CAPA installations #2934
   epic/logging provider/cluster-api-aws team/atlas +3
   
3. Add support for CAPA provider in logging-operator #2937
   epic/logging provider/cluster-api-aws team/atlas +3
   
4. Enable logging for all CAPA installations #2938
   epic/logging provider/cluster-api-aws team/atlas +3
   
5. Trust self-signed certificates in Promtail when installations use self-signed CAs #2956
   component/promtail epic/logging team/atlas +3
   

How to validate

• I can read logs from Loki using Grafana on each AWS installation

[QuentinBisson]

All items are deployed, let's check if we have logs on WCs

[TheoBrigitte]

Works on 2 out of 8 installations

Most of them return NetworkError from Grafana while trying to query for logs. goten has another issue as I can't reach out to Grafana.

• anzu 🔴
  
• gazelle 🟢
• goat 🔴
  
• golem 🟢
• goten 🔴
  
• grizzly 🔴
  
• velvet 🔴
  
• snail 🔴
  

[QuentinBisson]

I fixed the Root cause (incorrect cilium network policy for object-storage-operator). I'll release later. Goten IS ephemeral (mc-bootstrap running,it's not a Real installation)

[QuentinBisson]

Sée giantswarm/object-storage-operator#38

[QuentinBisson]

Currently only failing on Goat and Anzu, will investigate later

[QuentinBisson]

Anzu is fixed now.

Goat is working but promtail is failing becuase certificate is not secured:
level=warn ts=2023-11-13T16:50:41.225750028Z caller=client.go:419 component=client host=loki.goat.gaws.gigantic.io msg="error sending batch, will retry" status=-1 tenant=goat error="Post "https://loki.goat.gaws.gigantic.io/loki/api/v1/push\": tls: failed to verify certificate: x509: certificate is valid for ingress.local, not loki.goat.gaws.gigantic.io"

[QuentinBisson]

Issue for Goat:

#2956

We had this for prometheus agent as well.

@TheoBrigitte should we close this one (i'll let you validate of course) and keep the goat issue only?

[QuentinBisson]

Goat is fixed when using giantswarm/logging-operator#87

[QuentinBisson]

Loki is now running on CAPA, let's announce that