KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it #5033

mochizuki875 · 2025-01-10T05:01:01Z

One-line PR description: As discussed #129480, I've updated this KEP related to
labelSelector.
Issue link: Respect PodTopologySpread after rolling upgrades #3243
Other comments:

k8s-ci-robot · 2025-01-10T05:01:08Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mochizuki875
Once this PR has been reviewed and has the lgtm label, please assign ahg-g for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-scheduling/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

mochizuki875 · 2025-01-10T05:01:56Z

/cc @sanposhiho

sanposhiho · 2025-01-10T05:36:57Z

/cc @alculquicondor
Do we need /lead-opt-in at #3243?

@wojtek-t (or @alculquicondor) do we need PRR review in this case? (I suppose Yes?)

sanposhiho · 2025-01-10T05:37:48Z

/retitle KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it

alculquicondor · 2025-01-10T14:26:55Z

cc @dom4ha

sanposhiho

Update the content based on the conclusion kubernetes/kubernetes#129480 (comment)

sanposhiho · 2025-01-11T02:22:41Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

-existing pods over which spreading will be calculated.
-
-A new field named `MatchLabelKeys` will be introduced to`TopologySpreadConstraint`:
+A new optional field named `MatchLabelKeys` will be introduced to`TopologySpreadConstraint`.


We should keep this part.

Suggested change

A new optional field named `MatchLabelKeys` will be introduced to`TopologySpreadConstraint`.

A new optional field named `MatchLabelKeys` will be introduced to`TopologySpreadConstraint`.

Currently, when scheduling a pod, the `LabelSelector` defined in the pod is used

to identify the group of pods over which spreading will be calculated.

`MatchLabelKeys` adds another constraint to how this group of pods is identified

sanposhiho

You need to update other sections such as Test Plan and other PRR questions.

sanposhiho

Also, please add the current implementation for Alternative section and describe why we decided to move to a new approach.

mochizuki875 · 2025-01-14T08:54:56Z

I appreciate your comment.
I've addressed.

sanposhiho · 2025-01-15T05:24:06Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

 Use `pod.generateName` to distinguish new/old pods that belong to the 
 revisions of the same workload in scheduler plugin. It's decided not to 
 support because of the following reason: scheduler needs to ensure universal 
 and scheduler plugin shouldn't have special treatment for any labels/fields.

+### remove MatchLabelKeys implementation from the scheduler plugin


Suggested change

### remove MatchLabelKeys implementation from the scheduler plugin

### implement MatchLabelKeys in only either the scheduler plugin or kube-apiserver

Then, briefly mention why we have to implement it in kube-apiserver too.

sanposhiho · 2025-01-15T05:30:04Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

+kube-scheduler will also be aware of `matchLabelKeys` and gracefully handle the same labels.
+This is for the Cluster-level default constraints by 
+`matchLabelKeys: ["pod-template-hash"]`.([#129198](https://github.com/kubernetes/kubernetes/issues/129198))


Suggested change

kube-scheduler will also be aware of `matchLabelKeys` and gracefully handle the same labels.

This is for the Cluster-level default constraints by

`matchLabelKeys: ["pod-template-hash"]`.([#129198](https://github.com/kubernetes/kubernetes/issues/129198))

Also, kube-scheduler handles `matchLabelKeys` if the cluster-level default constraints is configured with `matchLabelKeys`.

sanposhiho · 2025-01-15T05:30:22Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

-disabled, the field `matchLabelKeys` is preserved if it was already set in the 
-persisted Pod object, otherwise it is silently dropped; moreover, kube-scheduler 
-will ignore the field and continue to behave as before.
+disabled, the field `matchLabelKeys` and corresponding`labelSelector` are preserved 


Suggested change

disabled, the field `matchLabelKeys` and corresponding`labelSelector` are preserved

disabled, the field `matchLabelKeys` and corresponding `labelSelector` are preserved

sanposhiho · 2025-01-15T05:31:05Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

+creation will be rejected by kube-apiserver; moreover, kube-scheduler will ignore the 
+field and continue to behave as before.


kube-scheduler cannot determine which label selector(s) is generated by matchLabelKeys at kube-apiserver, and hence it couldn't ignore matchLabelKeys even after the downgrade. the cluster-level default constraints configuration is the exception though

Suggested change

creation will be rejected by kube-apiserver; moreover, kube-scheduler will ignore the

field and continue to behave as before.

creation will be rejected by kube-apiserver.

Also, kube-scheduler will ignore matchLabelKeys in the cluster-level default constraints configuration.

sanposhiho · 2025-01-15T05:32:34Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

-In the event of a downgrade, kube-scheduler will ignore `MatchLabelKeys` even if it was set.
+In the event of a downgrade, kube-apiserver will reject pod creation with `matchLabelKeys` in `TopologySpreadConstraint`. 
+But, regarding existing pods, we leave `matchLabelKeys` and generated `LabelSelector` even after downgraded.
+kube-scheduler will ignore `MatchLabelKeys` even if it was set.


ditto

Suggested change

kube-scheduler will ignore `MatchLabelKeys` even if it was set.

kube-scheduler will ignore `MatchLabelKeys` if it was set in the cluster-level default constraints configuration.

sanposhiho · 2025-01-15T05:33:55Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

+disabling the feature gate, however kube-scheduler will not take the MatchLabelKeys 
+field into account.


Suggested change

disabling the feature gate, however kube-scheduler will not take the MatchLabelKeys

field into account.

disabling the feature gate.

sanposhiho · 2025-01-15T05:34:58Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

+kube-scheduler also looks up the label values from the pod and checks if those labels 
+are included in `LabelSelector`. If not, kube-scheduler will take those labels and AND 
+with `LabelSelector`.


Suggested change

kube-scheduler also looks up the label values from the pod and checks if those labels

are included in `LabelSelector`. If not, kube-scheduler will take those labels and AND

with `LabelSelector`.

kube-scheduler also handles matchLabelKeys if the cluster-level default constraints has it.

sanposhiho · 2025-01-15T05:36:31Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

+kube-scheduler will also look up the label values from the pod and check if those 
+labels are included in `LabelSelector`. If not, kube-scheduler will take those labels 
+and AND with `LabelSelector` to identify the group of existing pods over which the 
+spreading skew will be calculated.


Suggested change

kube-scheduler will also look up the label values from the pod and check if those

labels are included in `LabelSelector`. If not, kube-scheduler will take those labels

and AND with `LabelSelector` to identify the group of existing pods over which the

spreading skew will be calculated.

kube-scheduler will also handle it if the cluster-level default constraints have the one with `MatchLabelKeys`.

sanposhiho · 2025-01-15T05:38:16Z

keps/sig-scheduling/3243-respect-pod-topology-spread-after-rolling-upgrades/README.md

-which the spreading skew will be calculated.
+`TopologySpreadConstraint` which represent a set of label keys only. 
+kube-apiserver will use those keys to look up label values from the incoming pod 
+and those labels are merged to `LabelSelector`.


Suggested change

and those labels are merged to `LabelSelector`.

and those key-value labels are ANDed with `LabelSelector` to identify the group of existing pods over

which the spreading skew will be calculated.

sanposhiho · 2025-01-15T05:40:02Z

/assign @wojtek-t

for a PRR part reviewing. Please assign another person if needed.

wojtek-t · 2025-01-15T14:29:11Z

for a PRR part reviewing. Please assign another person if needed.

Queued - although I will wait for SIG approval to happen first.

update related to #129480

7a89964

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 10, 2025

k8s-ci-robot requested review from ahg-g and Huang-Wei January 10, 2025 05:01

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 10, 2025

k8s-ci-robot requested a review from sanposhiho January 10, 2025 05:01

mochizuki875 mentioned this pull request Jan 10, 2025

matchLabelKeys in PodTopologySpread should insert a labelSelector like PodAffinity's matchLabelKeys kubernetes/kubernetes#129480

Open

k8s-ci-robot requested a review from alculquicondor January 10, 2025 05:36

k8s-ci-robot changed the title ~~KEP-3243: Update description related to a labelSelector~~ KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it Jan 10, 2025

sanposhiho reviewed Jan 11, 2025

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 14, 2025

fix1

c6e0a76

mochizuki875 force-pushed the matchlabelkeys-to-podtopologyspread branch from 71f1c8c to c6e0a76 Compare January 14, 2025 09:04

sanposhiho reviewed Jan 15, 2025

View reviewed changes

k8s-ci-robot assigned wojtek-t Jan 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it #5033

KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it #5033

mochizuki875 commented Jan 10, 2025

k8s-ci-robot commented Jan 10, 2025

mochizuki875 commented Jan 10, 2025

sanposhiho commented Jan 10, 2025 •

edited

Loading

sanposhiho commented Jan 10, 2025

alculquicondor commented Jan 10, 2025

sanposhiho left a comment

sanposhiho Jan 11, 2025

sanposhiho left a comment

sanposhiho left a comment •

edited

Loading

mochizuki875 commented Jan 14, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho Jan 15, 2025

sanposhiho commented Jan 15, 2025

wojtek-t commented Jan 15, 2025

-A new optional field named `MatchLabelKeys` will be introduced to`TopologySpreadConstraint`.
+A new optional field named `MatchLabelKeys` will be introduced to`TopologySpreadConstraint`.
+Currently, when scheduling a pod, the `LabelSelector` defined in the pod is used
+to identify the group of pods over which spreading will be calculated.
+`MatchLabelKeys` adds another constraint to how this group of pods is identified

	### remove MatchLabelKeys implementation from the scheduler plugin
	### implement MatchLabelKeys in only either the scheduler plugin or kube-apiserver

	disabled, the field `matchLabelKeys` and corresponding`labelSelector` are preserved
	disabled, the field `matchLabelKeys` and corresponding `labelSelector` are preserved

		creation will be rejected by kube-apiserver; moreover, kube-scheduler will ignore the
		field and continue to behave as before.

	kube-scheduler will ignore `MatchLabelKeys` even if it was set.
	kube-scheduler will ignore `MatchLabelKeys` if it was set in the cluster-level default constraints configuration.

		disabling the feature gate, however kube-scheduler will not take the MatchLabelKeys
		field into account.

	disabling the feature gate, however kube-scheduler will not take the MatchLabelKeys
	field into account.
	disabling the feature gate.

	and those labels are merged to `LabelSelector`.
	and those key-value labels are ANDed with `LabelSelector` to identify the group of existing pods over
	which the spreading skew will be calculated.

KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it #5033

Are you sure you want to change the base?

KEP-3243: Update the design to mutate the label selector based on matchLabelKeys at api-server instead of the scheduler handling it #5033

Conversation

mochizuki875 commented Jan 10, 2025

k8s-ci-robot commented Jan 10, 2025

mochizuki875 commented Jan 10, 2025

sanposhiho commented Jan 10, 2025 • edited Loading

sanposhiho commented Jan 10, 2025

alculquicondor commented Jan 10, 2025

sanposhiho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanposhiho left a comment

Choose a reason for hiding this comment

sanposhiho left a comment • edited Loading

Choose a reason for hiding this comment

mochizuki875 commented Jan 14, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sanposhiho commented Jan 15, 2025

wojtek-t commented Jan 15, 2025

sanposhiho commented Jan 10, 2025 •

edited

Loading

sanposhiho left a comment •

edited

Loading