Unable to read the metrics through interceptors

[chandramohank]

Report

Insatlled the HttpAddOn and created the HttpScaler Object. However i am seeing the errors in Interceptor and external scaler Pod logs.

Http Scaler Object

kind: HTTPScaledObject
apiVersion: http.keda.sh/v1alpha1
metadata:
    name: webapp-demo-scaler
spec:
    targetPendingRequests: 10
    scaleTargetRef:
        deployment: webapp-demo
        service: webapp-demo
        port: 8080
    replicas:
        min: 1
        max: 3

Expected Behavior

Should not show any errors in Interceptor and external scaler pod logs. Should able to read the metrics without any erros.

Actual Behavior

Should able to retrieve the metrics without errors

Steps to Reproduce the Problem

Installed the HttpAddon on Azure AKS cluster and created the HttpScaler Object

Logs from KEDA HTTP operator

1. keda-add-ons-http-interceptor

{"level":"error","ts":1696737567.7369373,"logger":"Probe","caller":"handler/probe.go:63","msg":"health check function failed"
,"error":"table has not synced","errorVerbose":"table has not synced\ngithub.com/kedacore/http-add-on/pkg/routing.init\n\tgit
hub.com/kedacore/http-add-on/pkg/routing/table.go:25\nruntime.doInit\n\truntime/proc.go:6506\nruntime.doInit\n\truntime/proc.
go:6483\nruntime.doInit\n\truntime/proc.go:6483\nruntime.main\n\truntime/proc.go:233\nruntime.goexit\n\truntime/asm_amd64.s:1
598","stacktrace":"github.com/kedacore/http-add-on/interceptor/handler.(*Probe).check\n\tgithub.com/kedacore/http-add-on/inte
rceptor/handler/probe.go:63\ngithub.com/kedacore/http-add-on/interceptor/handler.(*Probe).Start\n\tgithub.com/kedacore/http-a
dd-on/interceptor/handler/probe.go:44"}

2. keda-add-ons-http-external-scaler

{"level":"error","ts":1696737812.986016,"logger":"GetMetricSpec","caller":"scaler/handlers.go:117","msg":"unable to get HTTPS
caledObject","name":"keda-add-ons-http-interceptor","namespace":"demo-cluster-ns","error":"httpscaledobject.http.keda.sh \"ke
da-add-ons-http-interceptor\" not found","stacktrace":"main.(*impl).GetMetricSpec\n\tgithub.com/kedacore/http-add-on/scaler/h
andlers.go:117\ngithub.com/kedacore/keda/v2/pkg/scalers/externalscaler._ExternalScaler_GetMetricSpec_Handler\n\tgithub.com/ke
dacore/keda/[email protected]/pkg/scalers/externalscaler/externalscaler_grpc.pb.go:190\ngoogle.golang.
org/grpc.(*Server).processUnaryRPC\n\tgoogle.golang.org/[email protected]/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleS
tream\n\tgoogle.golang.org/[email protected]/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\tgoogle.golan
g.org/[email protected]/server.go:966"}

HTTP Add-on Version

0.5.0

Kubernetes Version

1.26

Platform

Microsoft Azure

Anything else?

No response

[JorTurFer]

Hello,
The error was in the HTTP add-on code and chart. We released a new version (0.6.0) last week with this and other fixes, please give it a try and let me know if this has solved

[chandramohank]

Hi @JorTurFer

After upgraded to (0.6.0), the errors from keda-add-ons-http-external-scaler POD were gone. however i am still seeing the errors in keda-add-ons-http-interceptor POD.

{"level":"error","ts":1698145314.5528457,"logger":"Probe","caller":"handler/probe.go:63","msg":"health check function failed" ,"error":"table has not synced","stacktrace":"github.com/kedacore/http-add-on/interceptor/handler.(*Probe).check\n\tgithub.co m/kedacore/http-add-on/interceptor/handler/probe.go:63\ngithub.com/kedacore/http-add-on/interceptor/handler.(*Probe).Start\n\ tgithub.com/kedacore/http-add-on/interceptor/handler/probe.go:44"}

[JorTurFer]

Do you see those errors all the time? The error means that the internal table for routes hasn't been generated yet, but it should disappear after some seconds

[chandramohank]

I am seeing all the time.

…

On Wed, Oct 25, 2023, 03:01 Jorge Turrado Ferrero ***@***.***> wrote: Do you see those errors all the time? The error means that the internal table for routes hasn't been generated yet, but it should disappear after some seconds — Reply to this email directly, view it on GitHub <#812 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA4INBIUSTEVE3VJUQGY2ULYBAXTTAVCNFSM6AAAAAA5XM6TZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONZYGA3TGNBTGY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[JorTurFer]

Any idea @t0rr3sp3dr0 ?

[eumel8]

The current chart doesn't exposed a port for the metrics so there is no endpoint:

$ kubectl -n keda get ep keda-add-ons-http-controller-manager-metrics-service
NAME                                                   ENDPOINTS   AGE
keda-add-ons-http-controller-manager-metrics-service   <none>      4h39m

the operator deployment needs to extend:

template:
    spec:
      containers:
        ports:
        - containerPort: 8443
          name: https
          protocol: TCP

$ kubectl -n keda get ep keda-add-ons-http-controller-manager-metrics-service
NAME                                                   ENDPOINTS         AGE
keda-add-ons-http-controller-manager-metrics-service   10.42.1.89:8443   4h41m

[JorTurFer]

The service isn't correct, but it should point to metrics, we don't expose to https endpoint intentionally because we use a metrics proxy to protect the metrics through RBAC

[eumel8]

ah, ok. Just wondering this service had no endpoints

[JorTurFer]

ah, ok. Just wondering this service had no endpoints

Yeah, you are right and thanks to your comment I've noticed the gap. The PR is already opened 😄

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.