Help to transfer additional header to upstream service

[bkuchcik]

Hi everyone,

I need help, I have an issue to transfer header information to an upstream service with opa agent.

I have the following rego file:

authz if {
	is_token_valid
	action_allowed
}

headers["x-ext-auth-allow"] := "yes"
headers["x-validated-by"] := "security-checkpoint"
headers["x-should-show-only-data-on-domain"] := "?"

result["headers"] := headers

action_allowed if {
	http_request.method == "GET"
	role := token.payload.role
	acceptable_role(role)
	input.parsed_path == ["datasets"]
}

acceptable_role(role) {
  role == "guest"
}

acceptable_role(role) {
  role == "admin"
}

action_allowed if {
	http_request.method == "POST"
	token.payload.role == "admin"
	glob.match("/datasets", ["/"], http_request.path)
}

is_token_valid if {
	token.valid
	now := time.now_ns() / 1000000000
	token.payload.nbf <= now
	now < token.payload.exp
}

token := {"valid": valid, "payload": payload} if {
	[_, encoded] := split(http_request.headers.authorization, " ")
	[valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}

I want to test if I can add header to the current request for the upstream services.

I followed documentation here: https://www.openpolicyagent.org/docs/latest/envoy-primer/#output-document

I used the example on the opa website to configure envoy (with minor update):

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 8000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: service
          access_log:
            - name: envoy.access_loggers.stdout
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog               
          http_filters:
          - name: envoy.ext_authz
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
              transport_api_version: V3
              with_request_body:
                max_request_bytes: 131072
                allow_partial_message: false
              failure_mode_allow: false
              grpc_service:
                google_grpc:
                  target_uri: 127.0.0.1:9191
                  stat_prefix: ext_authz
                timeout: 0.5s
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router            
  clusters:
  - name: service
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: round_robin
    load_assignment:
      cluster_name: service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 8080
admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9901
      
layered_runtime:
  layers:
    - name: static_layer_0
      static_layer:
        envoy:
          resource_limits:
            listener:
              example_listener_name:
                connection_limit: 10000
        overload:
          global_downstream_max_connections: 50000

I just did minor adjustment.

Policy are evaluated, and I have the expected return type: in the context of a get request with a valid jwt token request is allowed. But when I watch the access log of the upstream services I see that headers properties (here "x-ext-auth-allow" for example) are not added to the request.

I tried to log in debug but I don't see the whole structure in the file:

{"decision":true,"dry-run":false,"err":null,"level":"debug","metrics":{"timer_rego_query_eval_ns":352384,"timer_server_handler_ns":0},"msg":"Returning policy decision.","query":"data.envoy.authz.authz","time":"2023-06-16T12:19:23Z","total_decision_time":1063287,"txn":35}
{"bundles":{"default":{}},"decision_id":"eb893afa-5c49-4478-bed5-98499b5f02ce","input":{"attributes":{"destination":{"address":{"socketAddress":{"address":"10.244.0.10","portValue":8000}}},"metadataContext":{},"request":{"http":{"headers":{":authority":"localhost",":method":"GET",":path":"/datasets?owner=Joe",":scheme":"http","accept":"*/*","authorization":"Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNDEwODE1MzksIm5iZiI6MTUxNDg1MTEzOSwicm9sZSI6Imd1ZXN0IiwiZG9tYWluZSI6InBlZGFudGUiLCJzdWIiOiJZV3hwWTJVPSJ9.iSfBq9Q6cEn5_VPjLoVTe3Z_Hx70SbBhpCsjPy677vc","user-agent":"curl/7.81.0","x-envoy-internal":"true","x-forwarded-for":"172.22.0.1","x-forwarded-host":"localhost","x-forwarded-port":"80","x-forwarded-proto":"http","x-forwarded-scheme":"http","x-real-ip":"172.22.0.1","x-request-id":"a088a86f373f72aa1d03a83daf83a380","x-scheme":"http"},"host":"localhost","id":"16481464339303866381","method":"GET","path":"/datasets?owner=Joe","protocol":"HTTP/1.1","scheme":"http"},"time":"2023-06-16T12:19:23.625972Z"},"source":{"address":{"socketAddress":{"address":"10.244.0.7","portValue":54248}}}},"parsed_body":null,"parsed_path":["datasets"],"parsed_query":{"owner":["Joe"]},"truncated_body":false,"version":{"encoding":"protojson","ext_authz":"v3"}},"labels":{"id":"fe312391-7c39-4a36-80ea-998d5224dce3","version":"0.53.1-envoy-2"},"level":"info","metrics":{"timer_rego_query_eval_ns":352384,"timer_server_handler_ns":1089890},"msg":"Decision Log","path":"envoy/authz/authz","result":true,"time":"2023-06-16T12:19:23Z","timestamp":"2023-06-16T12:19:23.628916639Z","type":"openpolicyagent.org/decision_logs"}

I don't know where is my mistake.

Could you help to fix the configuration so I can add headers to the request?

[ashutosh-narkar]

@bkuchcik I would first unit test the policy and verify if the correct output is received. Then try the same input and policy on a live system. The headers rule will only send additional request headers to the upstream if the request is allowed. The decision log should show the policy evaluation result. If everything looks ok and still you don't see the headers in the upstream service, check the Envoy logs as well.

[bkuchcik]

I thank you for the suggestion. I already added a test but I did not check that headers was in the output.

test_headers_contains_x_ext_auth_allow if {
    headers["x-ext-auth-allow"] == "yes" with input as {...}
}

I added the test but it passed.

I added log in my previous comment but it was not correctly formatted (it's opa logs)

As you can see I don't get any information about the structure returned by the policy (only result: true):

{
  "bundles": {
    "default": {}
  },
  "decision_id": "eb893afa-5c49-4478-bed5-98499b5f02ce",
  "input": {
    "attributes": {
      "destination": {
        "address": {
          "socketAddress": {
            "address": "10.244.0.10",
            "portValue": 8000
          }
        }
      },
      "metadataContext": {},
      "request": {
        "http": {
          "headers": {
            ":authority": "localhost",
            ":method": "GET",
            ":path": "/datasets?owner=Joe",
            ":scheme": "http",
            "accept": "*/*",
            "authorization": "Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjIyNDEwODE1MzksIm5iZiI6MTUxNDg1MTEzOSwicm9sZSI6Imd1ZXN0IiwiZG9tYWluZSI6InBlZGFudGUiLCJzdWIiOiJZV3hwWTJVPSJ9.iSfBq9Q6cEn5_VPjLoVTe3Z_Hx70SbBhpCsjPy677vc",
            "user-agent": "curl/7.81.0",
            "x-envoy-internal": "true",
            "x-forwarded-for": "172.22.0.1",
            "x-forwarded-host": "localhost",
            "x-forwarded-port": "80",
            "x-forwarded-proto": "http",
            "x-forwarded-scheme": "http",
            "x-real-ip": "172.22.0.1",
            "x-request-id": "a088a86f373f72aa1d03a83daf83a380",
            "x-scheme": "http"
          },
          "host": "localhost",
          "id": "16481464339303866381",
          "method": "GET",
          "path": "/datasets?owner=Joe",
          "protocol": "HTTP/1.1",
          "scheme": "http"
        },
        "time": "2023-06-16T12:19:23.625972Z"
      },
      "source": {
        "address": {
          "socketAddress": {
            "address": "10.244.0.7",
            "portValue": 54248
          }
        }
      }
    },
    "parsed_body": null,
    "parsed_path": [
      "datasets"
    ],
    "parsed_query": {
      "owner": [
        "Joe"
      ]
    },
    "truncated_body": false,
    "version": {
      "encoding": "protojson",
      "ext_authz": "v3"
    }
  },
  "labels": {
    "id": "fe312391-7c39-4a36-80ea-998d5224dce3",
    "version": "0.53.1-envoy-2"
  },
  "level": "info",
  "metrics": {
    "timer_rego_query_eval_ns": 352384,
    "timer_server_handler_ns": 1089890
  },
  "msg": "Decision Log",
  "path": "envoy/authz/authz",
  "result": true,
  "time": "2023-06-16T12:19:23Z",
  "timestamp": "2023-06-16T12:19:23.628916639Z",
  "type": "openpolicyagent.org/decision_logs"
}

From envoy perpective, even when I set the log in debug I don't see anything intersting in the log. Just the access and some stats.
The 400 error is the issue expected when the header is not set (the application backend sends it).

[2023-06-19T07:24:46.980Z] "GET /datasets?owner=Joe HTTP/1.1" 400 - 0 99 9 5 "172.22.0.1" "curl/7.81.0" "8f74bf656e0eb61ef47829b7a73204aa" "localhost" "127.0.0.1:8080"
[2023-06-19 07:24:50.477][1][debug][main] [source/server/server.cc:265] flushing stats
[2023-06-19 07:24:55.461][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:354] dns resolution for 127.0.0.1 started
[2023-06-19 07:24:55.461][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:275] dns resolution for 127.0.0.1 completed with status 0
[2023-06-19 07:24:55.461][1][debug][upstream] [source/common/upstream/upstream_impl.cc:457] transport socket match, socket default selected for host with address 127.0.0.1:8080

[srenatus]

What you haven't shared is your config used with opa-envoy-plugin (or did I just miss it?) I suspect the configured decision isn't result but authz. So, the headers you set are never returned to opa-envoy-plugin for the policy query, and therefore not presented to Envoy either.

Also, your result rule looks insufficient -- it's only got a "headers" key, no "allowed", like required and mentioned in the docs

[bkuchcik]

You are right, I misconfigured the opa agent and I fixed by referencing the result structure:

          args:
            - "run"
            - "--server"
            - "--log-level=debug"
            - "--addr=localhost:8181"
            - "--diagnostic-addr=0.0.0.0:8283"
            - "--set=services.default.url=http://nginx-bundles-server.default.svc.cluster.local/bundles"
            - "--set=bundles.default.resource=bundle.tar.gz"
            - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
            - "--set=plugins.envoy_ext_authz_grpc.path=envoy/authz/result"
            - "--set=decision_logs.console=true"
            - "--set=status.console=true"
            - "--ignore=.*"             

Now it fixed my issues thank you very much, I appreciated your help