Cors errors with Grafana Agent

[nlamirault]

Hi, i would like to test Faro with Grafana Agent with this application.
I use a K3s cluster with nginx-ingress-controller deployed, and a Grafana Agent with Faro configuration:

...

faro.receiver "LABEL" {
  server {
    listen_address = "0.0.0.0"
    cors_allowed_origins = [
      "*",
    ]
  }

  output {
    logs   = [
      loki.process.faro_logfmt.receiver,
    ]
    traces = [
      otelcol.exporter.otlp.local.input,
    ]
  }
}

...

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-methods: GET, PUT, POST, DELETE, PATCH,
      OPTIONS
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/cors-max-age: "1728000"
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: grafana-agent-frontend-faro
    app.kubernetes.io/instance: grafana-agent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana-agent-frontend
    app.kubernetes.io/version: v0.37.4
    argocd.argoproj.io/instance: grafana-agent
    helm.sh/chart: grafana-agent-frontend-0.27.2
  name: agent-frontend-faro-external
  namespace: opentelemetry
spec:
  ingressClassName: nginx
  rules:
  - host: agent-frontend-faro.home.portefaix.xyz
    http:
      paths:
      - backend:
          service:
            name: grafana-agent-grafana-agent-frontend-faro
            port:
              number: 12347
        path: /
        pathType: Prefix

Pizza configuration:

apiVersion: v1
data:
  QUICKPIZZA_CONF_FARO_URL: http://agent-frontend-faro.home.portefaix.xyz
  QUICKPIZZA_LOG_LEVEL: debug
  QUICKPIZZA_OTLP_ENDPOINT: http://grafana-agent-grafana-agent-traces.opentelemetry.svc.cluster.local:4318
kind: ConfigMap
metadata:
  labels:
    app.k8s.io/name: quickpizza
    app.kubernetes.io/component: config
    app.kubernetes.io/instance: tracing-env
    argocd.argoproj.io/instance: pizza-dev
  name: tracing-env
  namespace: demo-dev

Pizza application is deployed and works fine. But in the browser console, i've got these logs:

pizza-dev.home.portefaix.xyz/:1 Access to fetch at 'http://agent-frontend-faro.home.portefaix.xyz/' from origin 'http://pizza-dev.home.portefaix.xyz' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
start.27e40fd3.js:1 
        
        
POST http://agent-frontend-faro.home.portefaix.xyz/ net::ERR_FAILED 405 (Method Not Allowed)
window.fetch @ start.27e40fd3.js:1
(anonymous) @ app.000e2920.js:55
r.<computed> @ app.000e2920.js:48
(anonymous) @ app.000e2920.js:32
invoke @ app.000e2920.js:42
run @ app.000e2920.js:42
e.with @ app.000e2920.js:32
e.with @ app.000e2920.js:32
(anonymous) @ app.000e2920.js:32
p @ app.000e2920.js:55
(anonymous) @ app.000e2920.js:32
(anonymous) @ app.000e2920.js:31
o @ app.000e2920.js:1
(anonymous) @ app.000e2920.js:1
p @ app.000e2920.js:55
add @ app.000e2920.js:1
(anonymous) @ app.000e2920.js:31
(anonymous) @ app.000e2920.js:31
p @ app.000e2920.js:55
Nu @ app.000e2920.js:31
send @ app.000e2920.js:31
f @ app.000e2920.js:4
flush @ app.000e2920.js:1
(anonymous) @ app.000e2920.js:1
_.<computed> @ app.000e2920.js:91
invokeTask @ app.000e2920.js:42
runTask @ app.000e2920.js:42
invokeTask @ app.000e2920.js:42
x.useG.invoke @ app.000e2920.js:42
l.args.<computed> @ app.000e2920.js:91
setInterval (async)
s @ app.000e2920.js:91
scheduleTask @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleMacroTask @ app.000e2920.js:42
li @ app.000e2920.js:48
(anonymous) @ app.000e2920.js:91
r.<computed> @ app.000e2920.js:48
start @ app.000e2920.js:1
cs @ app.000e2920.js:1
ds @ app.000e2920.js:5
xs @ app.000e2920.js:29
Mu @ app.000e2920.js:32
(anonymous) @ app.000e2920.js:109
invoke @ app.000e2920.js:42
run @ app.000e2920.js:42
(anonymous) @ app.000e2920.js:55
invokeTask @ app.000e2920.js:42
runTask @ app.000e2920.js:42
v @ app.000e2920.js:42
Promise.then (async)
I @ app.000e2920.js:42
N @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleMicroTask @ app.000e2920.js:42
Te @ app.000e2920.js:55
le @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
Promise.then (async)
(anonymous) @ app.000e2920.js:55
p @ app.000e2920.js:55
O.then @ app.000e2920.js:55
le @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:55
invokeTask @ app.000e2920.js:42
runTask @ app.000e2920.js:42
v @ app.000e2920.js:42
Promise.then (async)
I @ app.000e2920.js:42
N @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleTask @ app.000e2920.js:42
scheduleMicroTask @ app.000e2920.js:42
Te @ app.000e2920.js:55
le @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
(anonymous) @ app.000e2920.js:54
Promise.then (async)
(anonymous) @ app.000e2920.js:55
p @ app.000e2920.js:55
O.then @ app.000e2920.js:55
nE @ app.000e2920.js:109
(anonymous) @ app.000e2920.js:109
app.000e2920.js:5 Faro
 @grafana/faro-web-sdk:transport-fetch
 Failed sending payload to the receiver
 {meta: {…}, events: Array(2), measurements: Array(2)} TypeError: Failed to fetch
    at window.fetch (start.27e40fd3.js:1:1545)
    at app.000e2920.js:55:3164
    at r.<computed> (app.000e2920.js:48:3547)
    at app.000e2920.js:32:123537
    at u.invoke (app.000e2920.js:42:6431)
    at s.run (app.000e2920.js:42:1820)
    at e.with (app.000e2920.js:32:136111)
    at e.with (app.000e2920.js:32:12794)
    at app.000e2920.js:32:123455
    at new p (app.000e2920.js:55:2172)

How can i fix that CORS error ? Thanks.

[nadiamoe]

Hi @nlamirault,

Your setup looks alright to me at first glance, but I would suggest some additional troubleshooting steps to see what's going on. The No 'Access-Control-Allow-Origin' header is present seems to hint that the ingress config is not materializing as expected. I would suggest to try the following:

Check that you don't have any other ingress set up for that host:

• kubectl get ingresses -A -o wide | grep -ie agent-frontend-faro

Curl the faro endpoint from the same machine running the browser and see what headers are returned:

• curl -v http://agent-frontend-faro.home.portefaix.xyz
• Maybe try a POST as well curl -v -d '{}' http://agent-frontend-faro.home.portefaix.xyz

Check the logs for your ingress controller and verify that you can see the request you make with both curl and the browser is correctly received. nginx also logs the service and endpoint backing the request, you can check it matches grafana-agent-grafana-agent-frontend-faro.

In the Network tab of your browser's development tools, inspect the request Faro is doing. You won't be able to see the response (it is blocked by cors), but the request might be worth checking, in case it includes strange headers.

Right click that request, "Copy as curl", and try to issue it in the terminal. There you will be able to see the response with its headers.

If this does not lead you to the problem, feel free to paste the output of those commands here, and we'll do our best to help you diagnose the situation :)

[nlamirault]

Hi @roobre i migrate to Alloy.
I've got 2 ingress:

• one for Alloy UI (which works fine)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-headers: DNT,X-CustomHeader,X-LANG,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,X-Api-Key,X-Device-Id,Access-Control-Allow-Origin
    nginx.ingress.kubernetes.io/cors-allow-methods: PUT, GET, POST, OPTIONS, DELETE
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: alloy-frontend
    app.kubernetes.io/instance: alloy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: alloy-frontend
    app.kubernetes.io/part-of: alloy
    app.kubernetes.io/version: v1.1.0
    argocd.argoproj.io/instance: alloy
    helm.sh/chart: alloy-frontend-0.3.0
    portefaix.xyz/version: v0.54.0
  name: alloy-frontend-external
  namespace: opentelemetry
spec:
  ingressClassName: nginx
  rules:
  - host: alloy-frontend.home.portefaix.xyz
    http:
      paths:
      - backend:
          service:
            name: alloy-alloy-frontend
            port:
              number: 12345
        path: /
        pathType: Prefix

• one for Alloy with Faro (on 12347)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/cors-allow-methods: GET, PUT, POST, DELETE, PATCH,
      OPTIONS
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/cors-max-age: "1728000"
    nginx.ingress.kubernetes.io/enable-cors: "true"
  labels:
    app: alloy-frontend
    app.kubernetes.io/instance: alloy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: alloy-frontend
    app.kubernetes.io/part-of: alloy
    app.kubernetes.io/version: v1.1.0
    argocd.argoproj.io/instance: alloy
    helm.sh/chart: alloy-frontend-0.3.0
    portefaix.xyz/version: v0.54.0
  name: alloy-frontend-faro-external
  namespace: opentelemetry
spec:
  ingressClassName: nginx
  rules:
  - host: alloy-frontend-faro.home.portefaix.xyz
    http:
      paths:
      - backend:
          service:
            name: alloy-alloy-frontend
            port:
              number: 12347
        path: /
        pathType: Prefix

and I've got an error using curl:

$ curl -v -d '{}' http://alloy-frontend-faro.home.portefaix.xyz
* Host alloy-frontend-faro.home.portefaix.xyz:80 was resolved.
* IPv6: (none)
* IPv4: 100.126.241.86
*   Trying 100.126.241.86:80...
* Connected to alloy-frontend-faro.home.portefaix.xyz (100.126.241.86) port 80
> POST / HTTP/1.1
> Host: alloy-frontend-faro.home.portefaix.xyz
> User-Agent: curl/8.6.0
> Accept: */*
> Content-Length: 2
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 405 Method Not Allowed
< Date: Fri, 21 Jun 2024 14:52:14 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 18
< Connection: keep-alive
< Allow: GET, OPTIONS
<
* Connection #0 to host alloy-frontend-faro.home.portefaix.xyz left intact
Method Not Allowed%

[nadiamoe]

Hey,

To me something seems to be off with your ingress-nginx annotations, or config. The curl command you posted should have returned CORS headers, something like:

< access-control-allow-origin: *
< access-control-allow-methods: GET, PUT, POST, DELETE, PATCH, OPTIONS
< access-control-allow-headers: DNT,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization

I'd suggest taking a look at the ingress-nginx manuals, logs, and/or generated config to try and figure out what's happening in there.