Autoscaling is not working on version v1.0.8 and kubectl 1.19.6

[thanos1983]

Is this a BUG REPORT or FEATURE REQUEST?:

What happened:
Autoscaling is not working. I am deploying 3 pods as minimum and 4 as maximum. The next second I list autoscaling and it is empty. I list pods and I only see 1

Sample:

$ kubeless autoscale create test --min 3 --max 4  --metric qps --value 100
INFO[0001] Adding autoscaling rule to the function...
INFO[0001] Autoscaling rule for test submitted for deployment
$ kubeless autoscale list
NAME    NAMESPACE       TARGET  MIN     MAX     METRIC  VALUE
$ kubeless function ls -n namespace-green
NAME    NAMESPACE       HANDLER         RUNTIME         DEPENDENCIES    STATUS
test    default test.foo        python3.8                       1/1 READY

What you expected to happen:
I would expect to see the number of pods to be equal to minimum int from the autoscale create.

How to reproduce it (as minimally and precisely as possible):
Simply create a function test or even from the documentation get-python and run the following:

$ kubeless autoscale create test --min 3 --max 4  --metric qps --value 100
INFO[0001] Adding autoscaling rule to the function...
INFO[0001] Autoscaling rule for test submitted for deployment
$ kubeless autoscale list
NAME    NAMESPACE       TARGET  MIN     MAX     METRIC  VALUE
$ kubeless function ls -n namespace-green
NAME    NAMESPACE       HANDLER         RUNTIME         DEPENDENCIES    STATUS
test    default test.foo        python3.8                       1/1 READY

Anything else we need to know?:
None

Environment:

• Kubernetes version (use kubectl version): Client Version: v1.19.6 / Server Version: v1.19.6
• Kubeless version (use kubeless version): Kubeless version: v1.0.8
• Cloud provider or physical cluster: Physical cluster

[andresmgot]

Hi @thanos1983,

Kubernetes version > 1.18 are not fully supported by Kubeless. See #1130 for more info.

Regarding autoscaling, Kubeless is creating an HorizontalPodAutoscaler:

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

As far as I can see, the HPA resource has not been moved to a different version, so it should keep working. Can you check if the HPA object has been created?
Also, note that for using metrics like qps you may need to install a metrics sever.

[thanos1983]

Hi @thanos1983,

Kubernetes version > 1.18 are not fully supported by Kubeless. See #1130 for more info.

Regarding autoscaling, Kubeless is creating an HorizontalPodAutoscaler:

https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

As far as I can see, the HPA resource has not been moved to a different version, so it should keep working. Can you check if the HPA object has been created?
Also, note that for using metrics like qps you may need to install a metrics sever.

Hello @andresmgot,

Thank you for getting back to me so fast.

For me on my cluster kubeless is running perfectly fine for function call and also REST calls.

The problem just appeared when I tried to autoscale the pods.

Regarding your question, I run metrics-server and also cri-o metrics server. See sample below:

$ kubectl get pods -A | grep metrics
cri-o-metrics-exporter   cri-o-metrics-exporter-6769b47f4f-jprtf        1/1     Running   1          18d
kube-system              metrics-server-7dd6f575fd-d955b                1/1     Running   918        18d
kubernetes-dashboard     dashboard-metrics-scraper-5f644f6df-hgfrb      1/1     Running   0          18d

Sample of function call:

$ kubeless function call test
Hello world!

Sample of REST function call:

$ curl https://example.my.domain.com
Hello world!

I will update my cluster tomorrow to 1.20.1 I will let you know if it works also there.

But is there any other way to see why the autoscaling is not working? Since the function and REST calls are working I would expect that autoscaling also to work.

[andresmgot]

Autoscaling is in a different API group, do you get anything if you list them: kubectl get hpa -o yaml?

[thanos1983]

kubectl get hpa -o yaml

No I am more confused, maybe because I am a beginner on autoscaling...

$ kubeless autoscale create test --min 3 --max 4  --metric cpu --value 70
INFO[0000] Adding autoscaling rule to the function...
INFO[0000] Autoscaling rule for test submitted for deployment
$ kubectl get hpa -o yaml
apiVersion: v1
items:
- apiVersion: autoscaling/v1
  kind: HorizontalPodAutoscaler
  metadata:
    creationTimestamp: "2021-02-09T10:48:19Z"
    managedFields:
    - apiVersion: autoscaling/v2beta1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:ownerReferences:
            .: {}
            k:{"uid":"d64fc7e0-0429-497b-a7be-387544b24f84"}:
              .: {}
              f:apiVersion: {}
              f:kind: {}
              f:name: {}
              f:uid: {}
        f:spec:
          f:maxReplicas: {}
          f:metrics: {}
          f:minReplicas: {}
          f:scaleTargetRef:
            f:apiVersion: {}
            f:kind: {}
            f:name: {}
      manager: kubeless-function-controller
      operation: Update
      time: "2021-02-09T10:48:19Z"
    name: test
    namespace: default
    ownerReferences:
    - apiVersion: kubeless.io/v1beta1
      kind: Function
      name: test
      uid: d64fc7e0-0429-497b-a7be-387544b24f84
    resourceVersion: "4837"
    uid: 54a4aa3a-d68e-4eed-a5eb-847f933d7fc0
  spec:
    maxReplicas: 4
    minReplicas: 3
    scaleTargetRef:
      apiVersion: apps/v1beta1
      kind: Deployment
      name: test
    targetCPUUtilizationPercentage: 70
  status:
    currentReplicas: 0
    desiredReplicas: 0
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
$ kubeless autoscale list
NAME    NAMESPACE       TARGET  MIN     MAX     METRIC  VALUE

Hope this gives you more information.

[andresmgot]

It seems that you are mixing namespaces. Why is the hpa in the namespace namespace-green? It should be in the namespace default (assuming the function test is deployed in that namespace).

[thanos1983]

It seems that you are mixing namespaces. Why is the hpa in the namespace namespace-green? It should be in the namespace default (assuming the function test is deployed in that namespace).

Yes you are absolutely right. Sorry I mixed those two by accident.

I created a namespace to separate it from default.

I just noticed that the pods are created in autoscaling but are not listed when I query the list:

$ kubeless autoscale list
NAME    NAMESPACE       TARGET  MIN     MAX     METRIC  VALUE

[andresmgot]

that may be a bug of the list command. Just to confirm, you are able to see the hpa if you do kubectl get hpa, right?

[thanos1983]

that may be a bug of the list command. Just to confirm, you are able to see the hpa if you do kubectl get hpa, right?

Hello again @andresmgot ,

Apologies for so late reply but I got stuck with something else.

Regarding your question:

$ kubectl get hpa
NAME   REFERENCE         TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
test   Deployment/test   <unknown>/70%   3         4         3          89s
$ kubeless autoscale list
NAME    NAMESPACE       TARGET  MIN     MAX     METRIC  VALUE

So yes it seems that hpa works but list does not.

Let me know if you need more information from my side.