Running Grafana Agent behind nginx at a sub path.

[ckpearson]

I'm currently working on getting a very minimal Grafana environment stood up and I have everything working (Loki, tempo, Mimir, Grafana and the agent as a collector).

I'm now trying to put it all behind nginx so I only have to expose the single route in.

I've got Grafana working under /grafana, and that was easy enough as the docs show how to get that done.
I've got the agent collection endpoints for traces, logs and metrics under /traces, /logs and /metrics respectively, that all works fine too.

The agent UI is what's giving me trouble.

I've got it under /agent, and after fiddling around and adding locations for the /public and /api that proxy_pass to the correct route those assets and endpoints work, but now I'm having to sub_filter the javascript to muck around with the routing and links.

I've searched the docs and the internet for how to get the agent running behind a reverse proxy but I can't find a thing.

Anyone done this / have any advice?

This is my nginx config:

resolver 127.0.0.11 ipv6=off;

# needed to proxy Grafana Live WebSocket connections
map $http_upgrade $connection_upgrade {
    default upgrade;
    '' close;
}

upstream grafana {
    server grafana:3000;
}

upstream agent-ui {
    server grafana-agent:12345;
}

upstream collect {
    server grafana-agent:4318;
}

server {
    listen 80;
    
    location /grafana {
        proxy_pass http://grafana;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;
    }

    location /grafana/api/live {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $http_host;
        proxy_pass http://grafana;
    }

    location /traces {
        proxy_pass http://collect/v1/traces;
    }

    location /metrics {
        proxy_pass http://collect/v1/metrics;
    }

    location /logs {
        proxy_pass http://collect/v1/logs;
    }

    location /agent {
        proxy_pass http://agent-ui;
        sub_filter 'href="/(.*)"' 'href="/agent/$1';
    }

    location ~ /public(.*) {
        proxy_set_header Accept-Encoding "";
        sub_filter_once off;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header XX-Forwarded-Proto $scheme;
        proxy_pass http://agent-ui/public$1;
        sub_filter "/graph" "/agent/graph";
        sub_filter 'path:"/"' 'path:"/agent"';
        sub_filter 'to:"/"' 'to:"/agent"';
        sub_filter_types *;
    }

    location ~ /api(.*) {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real_IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header XX-Forwarded-Proto $scheme;
        proxy_pass http://agent-ui/api$1;
    }
}

[rfratto]

We've been struggling with this internally too; after a few of our Javascript dependencies have been updated, we have to manually add a / at the end of the path for the UI to load properly when it's behind an Nginx proxy.

We spent a little bit of time looking into this, but other than knowing it's an issue with one of our dependencies, we haven't had enough time to fully investigate and fix the problem yet. @thampiotr did the original investigations (IIRC) and may have more information for what the problem was.