Manually scale Pod Autoscaler of a revision

[saileshd1402]

Proposed Changes

Update revision resource reconciler such that change in Revision annotations are reflected in it's Pod Autoscaler. With this change, we will be able to manually set the min-scale/max-scale of a PA by updating the annotations of a revision.

This will give us more control to manually scale up and down pods of a revision if needed.

Example:
Edit the annotations autoscaling.knative.dev/max-scale and autoscaling.knative.dev/min-scale of revision to update the Pod Autoscaler as well

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: saileshd1402
Once this PR has been reviewed and has the lgtm label, please assign davidhadas for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[linux-foundation-easycla]

• ✅login: saileshd1402 / (20fcdf3)

The committers listed above are authorized under a signed CLA.

[knative-prow]

Welcome @saileshd1402! It looks like this is your first PR to knative/serving 🎉

[knative-prow]

Hi @saileshd1402. Thanks for your PR.

I'm waiting for a knative member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[skonto]

Hi @saileshd1402 I suppose you want to avoid creating a new revision, what is the use case you have? Could you describe the problem that you have?

[saileshd1402]

Hi @skonto, essentially we are facing issues with manually scaling/updating replicaset. By default when we change scale, knative-serving brings up the pods associated with the new revision, wait for them to be active then scales down the previous revision. This doesn't work for on prem scenarios since there are fixed limited resources especially when dealing with GPUs so the new revision never gets ready.

[skonto]

Hi @saileshd1402 this is something https://github.com/knative-extensions/serving-progressive-rollout tries to address. You can ask @houshengbo on this one, they are facing the same in their org. Vincent maintains the extension and has some ideas on the topic.

[saileshd1402]

We did try using progressive rollout extension but it had it's own issues we faced:

1. Can there be an approach to terminate the last pod of the previous revision when most pods of new revision is up knative-extensions/serving-progressive-rollout#200
2. With the resourceUtil strategy, graceful traffic transfer does not occur during consecutive update requests. knative-extensions/serving-progressive-rollout#202
3. With the resourceUtil strategy, traffic transfer does not occur during an update when deployment hits quota limit knative-extensions/serving-progressive-rollout#203

In a brief summary, we have noticed that the "resourceUtil" strategy fails to do graceful traffic transfer during consecutive updates, particularly when resource limits are hit. This leads to stuck states and failed requests, as traffic may still continue to direct to terminated revisions while new pods remain in pending state.

[github-actions]

This Pull Request is stale because it has been open for 90 days with
no activity. It will automatically close after 30 more days of
inactivity. Reopen with /reopen. Mark as fresh by adding the
comment /remove-lifecycle stale.

[dprotaso]

Release is in a week - let's revisit this after

[skonto]

/remove-lifecycle stale

[elijah-rou]

I'm interested to see what it would take to enable this properly? I'd also like to be able to change scaling bounds and properties associated with the PodAutoscaler without minting a new revision. Currently it is being done through manipulating the PodAutoscaler CRD directly. So far in practice it doesn't seem to get reconciled but wondering what consequences this would have down the line

[dprotaso]

/ok-to-test

I think my only concern is what @elijah-rou brings up

I'd also like to be able to change scaling bounds and properties associated with the PodAutoscaler without minting a new revision. Currently it is being done through manipulating the PodAutoscaler CRD directly.

@elijah-rou Which fields are you changing? Is it just the annotations?

So far in practice it doesn't seem to get reconciled but wondering what consequences this would have down the line

The change in this PR would mean you'd have to manipulate the revision annotations instead of the PodAutoscaler ones.

[dprotaso]

I've sent an email to our users mailing list source interest/feedback on this - https://groups.google.com/g/knative-users/c/aEzUIwOK-_Y

[houshengbo]

I have a few questions about configurations directly with the revisions:

Question 1 : If the users would like to change the number of min or max replicas for the knative service for the latest version, why not change them via the knative service??

Question 2: If the users would like to change the number of min or max replicas for the knative service of an older version, what is the significance if the older versions/revisions do not exist any more?

[dprotaso]

Question 1 : If the users would like to change the number of min or max replicas for the knative service for the latest version, why not change them via the knative service??

This is a good question - cause effectively if you currently were to change the latest created revision then i believe the configuration reconciler would override these changes.

Question 2: If the users would like to change the number of min or max replicas for the knative service of an older version, what is the significance if the older versions/revisions do not exist any more?

An example could be you're using traffic splitting and want to rollback to an earlier revision and want to adjust that scale. Is that what your second question was asking?

[github-actions]

This Pull Request is stale because it has been open for 90 days with
no activity. It will automatically close after 30 more days of
inactivity. Reopen with /reopen. Mark as fresh by adding the
comment /remove-lifecycle stale.

[elijah-rou]

@dprotaso sorry, I missed your question from a while back.

I have just been updating the min_replicas, max_replicas and target annotations on the PodAutoscaler resource.

To give some more context to at least our use case; we have several users that wish to adjust scaling to the currently deployed revision. We however want to do this without minting a brand new revision (since this potentially requires the new revision to achieve new scale which we would have to provision; at least doubling the capacity when we wouldn't need to since the actual application has not seen any mutable changes).

It would be cool if we could make changes to the Knative Service directly (though I am happy on the Revision/PodAutoscaler as long as it is supported.) In my mind, changes to the annotations of a ksvc/revision affect configuration external to the deployed application and therefore should not mint a new revision (though this is purely subjective).

I don't know if @saileshd1402 is still working on this, but I could perhaps take this on. @skonto @dprotaso do you have any opinions on how this should not work? (ie what behaviour should this NOT change? eg my previous paragraph)

[dprotaso]

@elijah-rou

It would be cool if we could make changes to the Knative Service directly

I've been thinking about this but it would require significant changes in the autoscaler. It would essentially be min/max scale over all revisions. So that would be a longer thing with a feature track etc.

(though I am happy on the Revision/PodAutoscaler as long as it is supported.)

I think manipulating the Revision annotations makes the most sense. I think we just need to check that our logic to stamp out new revisions (when the config spec changes) doesn't break this by overwriting the existing revision