observability_mode is not working on ext_proc filter

[varkey98]

External Processing filter still waits for response from the server when observability_mode is set to true

Description:
According to the docs, when observability_mode is turned on, the filter shouldnt be waiting for responses from the server as the requests are just "Send and go". I've added a sleep in my server impl, and even with this mode turned on, my client gets the response only after my ext_proc server receives all parts of the request.

Repro steps:
Running on Mac OS Sonoma 14.5
Envoy Version: 816188b/1.30.1/Distribution/RELEASE/BoringSSL
Any server implementation of ext_proc filter which has a delay added and with observability_mode set to true will have this issue. Will upload my code somewhere and link it here for reference

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.

Config:

admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 9000


node:
  cluster: service_greeter
  id: test-id

static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          codec_type: AUTO
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/httpbin"
                route: 
                  cluster: service_httpbin
                  prefix_rewrite: "/"
              - match:
                  prefix: "/greet"
                route:
                  cluster: grpc_service_cluster
                  timeout: 0s
          http_filters:
          - name: envoy.filters.http.ext_proc
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
              failure_mode_allow: true
              allow_mode_override: false
              processing_mode:
                request_header_mode: "SEND"
                response_header_mode: "SEND"
                request_body_mode: "STREAMED"
                response_body_mode: "STREAMED"
                request_trailer_mode: "SEND"
                response_trailer_mode: "SEND"
              observability_mode: true
              message_timeout: 100s
              grpc_service:
                envoy_grpc:
                  cluster_name: ext_proc_cluster
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router  
  clusters:
  - name: service_httpbin
    connect_timeout: 2s
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: service_httpbin
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: httpbin.org
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
  - name: ext_proc_cluster
    type: STATIC
    connect_timeout: 1000s
    http2_protocol_options: {}
    load_assignment:
      cluster_name: ext_proc_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 127.0.0.1
                port_value: 18080
  - name: grpc_service_cluster
    type: STATIC
    connect_timeout: 1000s
    http2_protocol_options: { }
    load_assignment:
      cluster_name: ext_proc_cluster
      endpoints:
        - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    address: 127.0.0.1
                    port_value: 5067

Logs:

[2024-06-27 11:05:22.798][3458831][info][main] [source/server/server.cc:428] initializing epoch 0 (base id=0, hot restart version=disabled)
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:430] statically linked extensions:
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.common.key_value: envoy.key_value.file_based
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.network.dns_resolver: envoy.network.dns_resolver.apple, envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2024-06-27 11:05:22.799][3458831][info][main] [source/server/server.cc:432]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2024-06-27 11:05:22.800][3458831][info][main] [source/server/server.cc:432]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2024-06-27 11:05:22.800][3458831][info][main] [source/server/server.cc:432]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2024-06-27 11:05:22.800][3458831][info][main] [source/server/server.cc:432]   envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2024-06-27 11:05:22.800][3458831][info][main] [source/server/server.cc:432]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2024-06-27 11:05:22.801][3458831][info][main] [source/server/server.cc:432]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2024-06-27 11:05:22.801][3458831][info][main] [source/server/server.cc:432]   envoy.dubbo_proxy.protocols: dubbo
[2024-06-27 11:05:22.801][3458831][info][main] [source/server/server.cc:432]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2024-06-27 11:05:22.801][3458831][info][main] [source/server/server.cc:432]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   envoy.geoip_providers: envoy.geoip_providers.maxmind
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2024-06-27 11:05:22.802][3458831][info][main] [source/server/server.cc:432]   filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   network.connection.client: default, envoy_internal
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.string_matcher: envoy.string_matcher.lua
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.request_id: envoy.request_id.uuid
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2024-06-27 11:05:22.803][3458831][info][main] [source/server/server.cc:432]   envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.udp_packet_writer: envoy.udp_packet_writer.default
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.route_config_update_requester: envoy.route_config_update_requester.default
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.regex_engines: envoy.regex_engines.google_re2
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   quic.http_server_connection: quic.http_server_connection.default
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2024-06-27 11:05:22.804][3458831][info][main] [source/server/server.cc:432]   envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.http.injected_credentials: envoy.http.injected_credentials.generic
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.dynatrace
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.resolvers: envoy.ip
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.connection_handler: envoy.connection_handler.default
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment
[2024-06-27 11:05:22.805][3458831][info][main] [source/server/server.cc:432]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2024-06-27 11:05:22.806][3458831][info][main] [source/server/server.cc:432]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2024-06-27 11:05:22.811][3458831][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-06-27 11:05:22.811][3458831][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.cluster.v3.Cluster Using deprecated option 'envoy.config.cluster.v3.Cluster.http2_protocol_options' from file cluster.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-06-27 11:05:22.811][3458831][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-06-27 11:05:22.811][3458831][info][main] [source/server/server.cc:486] HTTP header map info:
[2024-06-27 11:05:22.813][3458831][info][main] [source/server/server.cc:489]   request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2024-06-27 11:05:22.813][3458831][info][main] [source/server/server.cc:489]   request trailer map: 120 bytes: 
[2024-06-27 11:05:22.813][3458831][info][main] [source/server/server.cc:489]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2024-06-27 11:05:22.814][3458831][info][main] [source/server/server.cc:489]   response trailer map: 144 bytes: grpc-message,grpc-status
[2024-06-27 11:05:22.822][3458831][info][main] [source/server/server.cc:861] runtime: {}
[2024-06-27 11:05:22.822][3458831][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.access_loggers.file.v3.FileAccessLog Using deprecated option 'envoy.extensions.access_loggers.file.v3.FileAccessLog.format' from file file.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-06-27 11:05:22.822][3458831][info][admin] [source/server/admin/admin.cc:66] admin address: 127.0.0.1:9000
[2024-06-27 11:05:22.822][3458831][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
[2024-06-27 11:05:22.822][3458831][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
[2024-06-27 11:05:22.822][3458831][info][config] [source/server/configuration_impl.cc:130] loading 3 cluster(s)
[2024-06-27 11:05:22.825][3458831][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
[2024-06-27 11:05:22.825][3458831][warning][config] [source/common/listener_manager/listener_impl.cc:1037] reuse_port was configured for TCP listener 'listener_0' and is being force disabled because Envoy is not running on Linux. See the documentation for more information.
[2024-06-27 11:05:22.827][3458831][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
[2024-06-27 11:05:22.827][3458831][warning][main] [source/server/server.cc:928] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.
[2024-06-27 11:05:22.827][3458831][info][main] [source/server/server.cc:969] starting main dispatch loop
[2024-06-27 11:05:22.827][3458831][info][runtime] [source/common/runtime/runtime_impl.cc:614] RTDS has finished initialization
[2024-06-27 11:05:22.827][3458831][info][upstream] [source/common/upstream/cluster_manager_impl.cc:240] cm init: all clusters initialized
[2024-06-27 11:05:22.827][3458831][info][main] [source/server/server.cc:950] all clusters initialized. initializing init manager
[2024-06-27 11:05:22.827][3458831][info][config] [source/common/listener_manager/listener_manager_impl.cc:930] all dependencies initialized. starting workers

[htuch]

Observability mode is still WiP, @tyxia

[varkey98]

Thank you @htuch for the prompt reply :)

[tyxia]

@varkey98 Hi. Thanks for trying it out.

The actual "send and go" mode are recently added in #34392 and #34491

It seems that your Envoy binary is release in April 18th which doesn't include actual implementation yet.

[varkey98]

Thanks @tyxia, let me try building it from main. Would there be a release soon with this feature?

[tyxia]

SG, let me know how is your testing going with actual ob11y support.

Regarding the release: It will be included in upcoming release https://www.envoyproxy.io/docs/envoy/latest/version_history/v1.31/v1.31.0

[varkey98]

I think the docs need an update. The docs say that everything except async mode is implemented and there's no mention about observability mode being the new async mode. Also in the configuratio, observability mode is mentioned without specifying that its not implemented yet. I'm guessing this would've made many like me try this out without realising it wasn't implemented