Log-Receiver: Implementation

[Rotfuks]

Motivation

In order to enable customers to receive logs also from outside the installations we have to get our hands dirty and implement the thing we created a concept for in the investigation story.

Todo

• Implement the solution discussed in the investigation story
• TODO // Add more details after investigation closed

Outcome

• We have a working log receiver which can receive logs from different data sources outside of the installations.

[QuentinBisson]

Coming from the investigation here #3567 the implementation now is to:

• Deploy an alloy instance to act as an open-telemetry collector (alloy-gateway) enabling OLTP http endpoint only for now (https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.otlp/). The ingress could be named oltp.observability.<installation base domain> for instance.
  The logs needs to have a tenant defined, otherwise they need to be rejected.
• Once the logs are received, the gateway sends the received logs to loki directly.
• We will probably have to create a datasource by hand for those logs though as they would not pass through the multi-tenant gateway.

Regarding security, after some discussions with Zach at the onsite, considering that we cannot have workload identity now, we could make sure the ingress uses OIDC with the customer's SSO so they would have to make sure their app as the permission to write to our endpoint :)

@giantswarm/team-atlas this is a lot easier than the original implementation and also more secure than API keys. Are you fine with it so we can start the implementation?

[QuentinBisson]

Waiting for initial PRs to be approved:

• Deploy the alloy gateway to CAPI MCs with a default config.

[QuentinBisson]

Pushing to collection manually because architect is broken: giantswarm/architect-orb#555

• push alloy-gateway to collection vsphere-app-collection#117
• push alloy-gateway to collection capa-app-collection#71
• push alloy-gateway to collection cloud-director-app-collection#135
• push alloy-gateway to collection capz-app-collection#112

[QuentinBisson]

Well, collections do not allow us to deploy an app with a different name https://gigantic.slack.com/archives/C02GDJJ68Q1/p1726752552645969

[QuentinBisson]

Blocked by #3682

[QuentinBisson]

Let's unblock us with this hack for now https://github.com/giantswarm/alloy-gateway-app

This application actually deploys an App Cr named observability-gateway which is actually an instance of Alloy the same way alloy-rules is deployed in prometheus-rules

[QuentinBisson]

Well, i'm blocked again https://gigantic.slack.com/archives/C02GDJJ68Q1/p1727272307578759 :D

[QuentinBisson]

Current configuration PR is here https://github.com/giantswarm/shared-configs/pull/158

This is being tested on grizzly but the X-Scope-OrgID header is currently not being picked up and I think it's because the feature is not built into alloy yet grafana/alloy#1805.

This does not really prevents us from enabling the gateway if we set a random tenant like external for all external logs using the following stage:

loki.process "tenant" {
    stage.tenant {
        value = "external"
    }
}

[QuentinBisson]

Alright so I managed to make this work by using I don't know how many hacks :D

Gateway configuration

    alloy:
      enabled: true
      alloy:
        configMap:
          create: true
          content: |
            loki.write "local" {
                endpoint {
                    url = "http://loki-gateway.loki.svc/loki/api/v1/push"
                }
            }

            loki.echo "example" { }

            loki.process "tenant" {
                forward_to = [
                    loki.echo.example.receiver,
                    loki.write.local.receiver,
                ]

                stage.tenant {
                    value = "external"
                }
            }

            loki.source.api "loki_push_api" {
                http {
                    listen_address = "0.0.0.0"
                    listen_port = 3100
                }
                use_incoming_timestamp = true
                forward_to = [
                    loki.process.tenant.receiver,
                ]
                labels = {
                  forwarded = "true",
                }
            }
        extraPorts:
        - name: "loki-api"
          port: 3100
          targetPort: 3100
          protocol: "TCP"

      controller:
        type: 'deployment'
        autoscaling:
          enabled: true
      # The gateway does not need pods logs
      crds:
        create: false
      ingress:
        enabled: true
        ingressClassName: nginx
        annotations:
          cert-manager.io/cluster-issuer: letsencrypt-giantswarm
          nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
          nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
          # Ensure requests have the X-Scope-OrgID header set
          nginx.ingress.kubernetes.io/configuration-snippet: |
            if ($http_x_scope_orgid = "") {
              return 401;
            }
            add_header X-Scope-OrgID $http_x_scope_orgid;
        hosts:
        - gateway.observability.grizzly.gaws.gigantic.io
        extraPaths:
        - path: /loki/api/v1/push
          pathType: Prefix
          backend:
            service:
              name: observability-gateway-alloy
              port:
                name: "loki-api"
        tls:
        - hosts:
          - gateway.observability.grizzly.gaws.gigantic.io
          secretName: tls-certificate-observability-gateway
    networkPolicy:
      cilium:
	egress:
        - toEntities:
          - kube-apiserver
          - cluster
        ingress:
        - fromEntities:
          - cluster
          - world

Oauth2-proxy

It needs to be redeployed ...

Config (I'm not sure what all those fields even do though)

  values: |
    oauth2Proxy:
      extraEnv:
      - name: 'OAUTH2_PROXY_EMAIL_DOMAINS'
        value: '*'
      - name: 'OAUTH2_PROXY_PROVIDER_DISPLAY_NAME'
        value: 'Dex'
      - name: 'OAUTH2_PROXY_SKIP_PROVIDER_BUTTON'
        value: 'true'
      - name: 'OAUTH2_PROXY_SKIP_JWT_BEARER_TOKENS'
        value: 'true'
      - name: 'OAUTH2_PROXY_SET_AUTHORIZATION_HEADER'
        value: 'true'
      - name: 'OAUTH2_PROXY_SET_XAUTHREQUEST'
        value: 'true'
      - name: 'OAUTH2_PROXY_PASS_ACCESS_TOKEN'
        value: 'true'
      - name: 'OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER'
        value: 'true'
      - name: 'OAUTH2_PROXY_COOKIE_NAME'
        value: SESSION
    ingress:
      enabled: true
      hosts:
        - gateway.observability.grizzly.gaws.gigantic.io

Dex

Dex configuration needs to have a new redirect URI configured for hte gateway, this will require some changes in mc-bootstrap and in configs to change the secret value ...

A little hack to be able to generate a token with dex https://mac-blog.org.ua/dex-between-services/ (edit the dex secret in the giantswarm namespace)

Alloy

Upstream PR has been merged but we will need to wait for the helm chart to be released upstream before we can really provide it.

Useful link: https://developer.okta.com/blog/2022/07/14/add-auth-to-any-app-with-oauth2-proxy

[Rotfuks]

Great Job! That was a beast of a story, huh?
Are we happy with the result, even when it's a bit hacky, or should we discuss if this actually meets our quality gate and if it's leading to lots of maintaining pain in the future?

[QuentinBisson]

I'm bringing this up this morning because I have doubts about oauth 😅 Le mar. 8 oct. 2024, 08:28, Dominik Kress ***@***.***> a écrit :

…

Great Job! That was a beast of a story, huh? Are we happy with the result, even when it's a bit hacky, or should we discuss if this actually meets our quality gate and if it's leading to lots of maintaining pain in the future? — Reply to this email directly, view it on GitHub <#3568 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVIRXEWE3UX7VYWKXJJCJDZ2N3RJAVCNFSM6AAAAABKU7G672VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJYHE2TQNZWHE> . You are receiving this because you were assigned.Message ID: ***@***.***>

[QuentinBisson]

Current wip:

• We are waiting for alloy 1.5.0 to be released to include our change to accept the org-id header
• https://github.com/giantswarm/mc-bootstrap/pull/1030 is blocking, this PR configures oauth2-proxy again to support dex for now

We are waiting on customer feedback to know if they would be able to support oauth. If they are, we will need to change the oauth2-proxy config towards the customer's OIDC provider @Rotfuks any news on this?

[QuentinBisson]

Alloy released upstream and in the gateway giantswarm/alloy-gateway-app#12 (review)