VPA: Implement in-place updates support #7673

maxcao13 · 2025-01-07T18:10:29Z

What type of PR is this?

/kind feature
/kind api-change

What this PR does / why we need it:

This PR is an attempt to implement VPA in-place vertical scaling according to AEP-4016. It uses the VPA updater to actuate recommendations by sending resize patch requests to pods which allows in-place resize as enabled by the InPlacePodVerticalScaling feature flag in k8s 1.27.0 alpha and above (or by eventual graduation).

It includes some e2e tests currently according to the AEP, but I am sure we will probably need more.

This PR is a continuation of #6652 started by @jkyros with a cleaner git commit history.

Which issue(s) this PR fixes:

Fixes #4016

Special notes for your reviewer:

Notable general areas of concern:

We just kind of hacked the in-place stuff into the eviction limiter, maybe it should have been its own thing, or maybe we need a "disruption limiter", but in-place and eviction needed to know about each other because they have the same "disruption limit"
For now, there are many many TODOs literred throughout the code which need attention from reviewers/maintainers. A lot is because of design decisions I probably shouldn't make on my own. I resolved some of John's TODOs but he still has relevant comments that need to be addressed as well. I am using the TODOs as the "special notes for your reviewer" section, if people would like a comment somewhere which lays them all out nicely, I'm more than happy to make one.
Requires a lot more unit testing, but if a lot of architecture is to change, I chose delay writing them until I get feedback.
There's additional comments by John which can help aid review in the earlier commit descriptions.

Does this PR introduce a user-facing change?

In-place VPA scaling implemented, it can be enabled by setting `updateMode` on your VPA to `InPlaceOrRecreate` (Depends on `InPlaceVerticalPodScaling` feature gate being enabled or having graduated)

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[AEP] https://github.com/kubernetes/autoscaler/tree/09954b6741cbb910971916c079f45f6e8878d192/vertical-pod-autoscaler/enhancements/4016-in-place-updates-support
Depends on: 
[KEP] https://github.com/kubernetes/enhancements/tree/25e53c93e4730146e4ae2f22d0599124d52d02e7/keps/sig-node/1287-in-place-update-pod-resources

k8s-ci-robot · 2025-01-07T18:10:38Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: maxcao13
Once this PR has been reviewed and has the lgtm label, please assign voelzmo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

vertical-pod-autoscaler/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-01-07T18:10:39Z

Hi @maxcao13. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-triage-robot · 2025-01-07T18:55:03Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

adrianmoisey · 2025-01-07T19:35:55Z

/ok-to-test

omerap12

This is just my first review after going through all the changes. I will go over it multiple times, but these are my initial comments for now.

vertical-pod-autoscaler/hack/run-e2e-locally.sh

omerap12 · 2025-01-10T08:39:25Z

vertical-pod-autoscaler/pkg/utils/annotations/vpa_inplace_update.go

+
+package annotations
+
+// TODO(maxcao13): This annotation currently doesn't do anything. Do we want an annotation to show vpa inplace resized only for cosmetic reasons?


I'm not a fan of embedding logic into annotations. So, +1 for making this change purely for cosmetic reasons.

vertical-pod-autoscaler/pkg/apis/autoscaling.k8s.io/v1/types.go

vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go

omerap12 · 2025-01-10T09:03:01Z

vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go

+					"evicted", singleGroupStats.evicted,
+					"updating", singleGroupStats.inPlaceUpdating)
+
+				if singleGroupStats.running-(singleGroupStats.evicted+(singleGroupStats.inPlaceUpdating-1)) > shouldBeAlive {


Can you explain this logic? why do we have singleGroupStats.inPlaceUpdating-1 and not just singleGroupStats.inPlaceUpdating? is it because that the pod is already in inPlaceUpdating phase? (because of this line: if IsInPlaceUpdating(pod) )

vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go

omerap12 · 2025-01-10T11:33:35Z

vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go

+	//result := []*apiv1.Pod{}
+	result := []*PrioritizedPod{}
+	for num, podPrio := range calc.pods {
+		if admission.Admit(podPrio.pod, podPrio.recommendation) {


Can you please explain what do we check here? I thought the purpose of this function is to sort pods based on priority

vertical-pod-autoscaler/pkg/updater/priority/update_priority_calculator.go

This just addes the UpdateModeInPlaceOrRecreate mode to the types so we can use it. I did not add InPlaceOnly, as that seemed contentious and it didn't seem like we had a good use case for it yet.

Signed-off-by: Max Cao <[email protected]>

So because of InPlacePodVerticalScaling, we can have a pod object whose resource spec is correct, but whose status is not, because that pod may have been updated in-place after the original admission. This would have been ignored until now because "the spec looks correct", but we need to take the status into account as well if a resize is in progress. This commit: - takes status resources into account for pods/containers that are being in-place resized - makes sure that any pods that are "stuck" in-place updating (i.e. the node doesn't have enough resources either temporarily or permanently) will still show up in the list as having "wrong" resources so they can still get queued for eviction and be re-assigned to nodes that do have enough resources

This commit makes the eviction restrictor in-place update aware. While this possibly could be a separate restrictor or refactored into a shared "disruption restrictor", I chose not to do that at this time. I don't think eviction/in-place update can be completely separate as they can both cause disruption (albeit in-place less so) -- they both need to factor in the total disruption -- so I just hacked the in-place update functions into the existing evictor and added some additional counters for disruption tracking. While we have the pod lifecycle to look at to figure out "where we are" in eviction, we don't have that luxury with in-place, so that's why we need the additional "IsInPlaceUpdating" helper.

The updater logic wasn't in-place aware, so I tried to make it so. The thought here is that we try to in-place update if we can, if we can't or if it gets stuck/can't satisfy the recommendation, then we fall back to eviction. I tried to keep the "blast radius" small by stuffing the in-place logic in its own function and then falling back to eviction if it's not possible. It would be nice if we had some sort of "can the node support an in-place resize with the current recommendation" but that seemed like a whole other can of worms and math.

We might want to add a few more that are combined disruption counters, e.g. in-place + eviction totals, but for now just add some separate counters to keep track of what in-place updates are doing.

For now, this just updates the mock with the new functions I added to the eviction interface. We need some in-place test cases.

TODO(jkyros): come back here and look at this after you get it working

So far this is just: - Make sure it scales when it can But we still need a bunch of other ones like - Test fallback to eviction - Test timeout/eviction when it gets stuck, etc

In the event that we can't perform the whole update, this calculates a set of updates that should be disruptionless and only queues that partial set, omitting the parts that would cause disruption.

Signed-off-by: Max Cao <[email protected]>

maxcao13 · 2025-01-15T02:27:29Z

Appreciate the review! I will respond to the other comments tomorrow, just wanted to get the easy stuff out of the way and less cluttered.

omerap12 · 2025-01-15T12:55:19Z

Appreciate the review! I will respond to the other comments tomorrow, just wanted to get the easy stuff out of the way and less cluttered.

Sure, take your time

omerap12 · 2025-01-15T13:48:42Z

vertical-pod-autoscaler/pkg/updater/eviction/pods_eviction_restriction.go

+			klog.V(4).InfoS("Patched pod annotations", "pod", klog.KObj(res), "patches", string(patch))
+		}
+	} else {
+		err := fmt.Errorf("no patches to apply to %s", podToUpdate.Name)


Can we combine it to one? Maybe something like that:

klog.ErrorS("Failed to patch pod. no patches to apply", "pod", klog.KObj(podToUpdate))

I'm not sure if that would work, I still need to return an err. Maybe this is better? WDYT?

err := fmt.Errorf("no resource patches were calculated to apply") klog.ErrorS(err, "Failed to patch pod", "pod", klog.KObj(podToUpdate)) return err

Signed-off-by: Max Cao <[email protected]>

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels Jan 7, 2025

k8s-ci-robot requested review from jbartosik and kwiesmueller January 7, 2025 18:10

k8s-ci-robot added area/vertical-pod-autoscaler needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jan 7, 2025

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 7, 2025

omerap12 reviewed Jan 10, 2025

View reviewed changes

jkyros and others added 14 commits January 14, 2025 08:56

VPA: Add UpdateModeInPlaceOrRecreate to types

b583049

This just addes the UpdateModeInPlaceOrRecreate mode to the types so we can use it. I did not add InPlaceOnly, as that seemed contentious and it didn't seem like we had a good use case for it yet.

VPA: Allow admission-controller to validate in-place spec

2b84bef

Signed-off-by: Max Cao <[email protected]>

VPA: Add metrics gauges for in-place updates

0236fb7

We might want to add a few more that are combined disruption counters, e.g. in-place + eviction totals, but for now just add some separate counters to keep track of what in-place updates are doing.

VPA: Update mocks to accommodate in-place VPA changes

38bd58e

For now, this just updates the mock with the new functions I added to the eviction interface. We need some in-place test cases.

VPA: hack unit tests to account for in-place

45f082a

TODO(jkyros): come back here and look at this after you get it working

VPA: Add e2e tests for in-place scaling

ff29419

So far this is just: - Make sure it scales when it can But we still need a bunch of other ones like - Test fallback to eviction - Test timeout/eviction when it gets stuck, etc

VPA: allow rule-breaking updates if disruptionless

d25a609

In the event that we can't perform the whole update, this calculates a set of updates that should be disruptionless and only queues that partial set, omitting the parts that would cause disruption.

only allow in-place if explicitly set (for testing)

e1ca639

VPA: Allow VPA updater to actuate recommendations in-place

aec718a

Signed-off-by: Max Cao <[email protected]>

VPA: Enable InPlacePodVerticalScaling feature flag on e2e

f2b8124

Signed-off-by: Max Cao <[email protected]>

VPA: Add in-place actuation and admission-controller e2e tests

e406979

Signed-off-by: Max Cao <[email protected]>

VPA: fix logs and cleanup TODOs according to review

53d4006

Signed-off-by: Max Cao <[email protected]>

maxcao13 force-pushed the maxcao13-inplace branch from 6ea7337 to 53d4006 Compare January 15, 2025 02:23

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 15, 2025

omerap12 reviewed Jan 15, 2025

View reviewed changes

maxcao13 added 2 commits January 15, 2025 10:11

fixup! VPA: Add UpdateModeInPlaceOrRecreate to types

0fdfd98

Signed-off-by: Max Cao <[email protected]>

fixup! VPA: fix logs and cleanup TODOs according to review

f4ad3ca

Signed-off-by: Max Cao <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VPA: Implement in-place updates support #7673

VPA: Implement in-place updates support #7673

maxcao13 commented Jan 7, 2025

k8s-ci-robot commented Jan 7, 2025

k8s-ci-robot commented Jan 7, 2025

k8s-triage-robot commented Jan 7, 2025

adrianmoisey commented Jan 7, 2025

omerap12 left a comment

omerap12 Jan 10, 2025

omerap12 Jan 10, 2025

omerap12 Jan 10, 2025

maxcao13 commented Jan 15, 2025

omerap12 commented Jan 15, 2025

omerap12 Jan 15, 2025

maxcao13 Jan 15, 2025


		package annotations

		// TODO(maxcao13): This annotation currently doesn't do anything. Do we want an annotation to show vpa inplace resized only for cosmetic reasons?

VPA: Implement in-place updates support #7673

Are you sure you want to change the base?

VPA: Implement in-place updates support #7673

Conversation

maxcao13 commented Jan 7, 2025

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Jan 7, 2025

k8s-ci-robot commented Jan 7, 2025

k8s-triage-robot commented Jan 7, 2025

adrianmoisey commented Jan 7, 2025

omerap12 left a comment

Choose a reason for hiding this comment

omerap12 Jan 10, 2025

Choose a reason for hiding this comment

omerap12 Jan 10, 2025

Choose a reason for hiding this comment

omerap12 Jan 10, 2025

Choose a reason for hiding this comment

maxcao13 commented Jan 15, 2025

omerap12 commented Jan 15, 2025

omerap12 Jan 15, 2025

Choose a reason for hiding this comment

maxcao13 Jan 15, 2025

Choose a reason for hiding this comment