KEP-4330: Change API availability to forward compatibility and add NonEmulatableFeatures #5038
Conversation
siyuanfoundation
commented
Jan 14, 2025
siyuanfoundation
commented
- One-line PR description: some feature updates to make it easier to handle API graduation in compatibility version.
- Issue link: Introduce compatibility version for Kubernetes control-plane upgrades #4330
- Other comments: more discussions can be found in [Compatibility Version] Guidance on API version promotion kubernetes#127791
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: siyuanfoundation The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
kind/kep
siyuanfoundation
force-pushed
the
compat-versions-api
branch
2 times, most recently
from
6abb021 to
fbbc5f7
Compare
| Instead, we are proposing to keep forward compatibility of all AIPs in compatibility version mode: | ||
| All APIs group-versions declarations would be modified | ||
| to track which Kubernetes version the API group-versions are removed at. APIs introduced between the emulation version and | ||
| the binary version are still available if the stability level is no less than the APIs introduced before or at the emulation version and enabled at the emulation version. |
There was a problem hiding this comment.
Most of this is referring to API enablement/disablement on a group-version level. However, API lifecycles are generally individual on the group-version-resource level. I think it would make more sense to discuss the individual beta->GA resources rather than the entire group version?
|
|
||
| Alpha APIs may not be enabled in conjunction with emulation version. | ||
|
|
||
| The forward compatibility of APIs should not affect data compatibility because storage version is still controlled by the `MinCompatibilityVersion` regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is `Equivalent`. |
There was a problem hiding this comment.
I think we should add a section on how MinCompatibilityVersion will change storage version since the policy has always been n-1.
There was a problem hiding this comment.
Updated in #5006
| to track which Kubernetes version the API group-versions are introduced (or | ||
| removed) at. | ||
| removed) at. But in practice, that would make code changes of API graduation intractable. |
There was a problem hiding this comment.
intractable only for withholding API introductions. API removals don't need special handling.
There was a problem hiding this comment.
Updated the whole section. Please review again see if it is more clear.
| @@ -610,21 +623,25 @@ The storage version of each group-resource is the newest | |||
|
|
|||
| ### API availability | |||
|
|
|||
| Similar to feature flags, all APIs group-versions declarations will be modified | |||
| Ideally, similar to feature flags, all APIs group-versions declarations should be modified | |||
There was a problem hiding this comment.
Is this referring to the prerelease lifecycle files or a different mechanism? prerelease lifecycle operates on the types instead of the group version.
There was a problem hiding this comment.
it is referring to the the prerelease lifecycle.
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
fbbc5f7 to
ebd2f55
Compare
| removed) at: if an API is introduced after (or removed before) the emulation version, it should not be available at the emulation version. But in practice, that would make code changes of API graduation that wants to use newer APIs intractable. | ||
| For example, if a controller is switching from Beta to GA API at the binary version, a full emulation would mean the whole controller code needs to be |
There was a problem hiding this comment.
My understanding is that we're doing this specifically for controllers. Are there other use cases or should we simplify the wording here and just state up front that we're doing this to support controllers specifically?
| the emulation version is set to. I.e. `--runtime-config=api/<version>` needs | ||
| to be able to turn on APIs exactly like it did for each Kubernetes version that | ||
| emulation version can be set to. | ||
| If an API has GAed and has not been removed at the emulated version, it would be enabled by default at the emulation version. If a newer stable version of the GA API has been introduced between the emulation version and the binary version, the new GA API would also be enabled at the emulation version along with the old GA API. |
There was a problem hiding this comment.
A comparison (maybe in a table), or how emulation works for normal APIs vs how it works for un-emulatable (non-emulatable?) features might make this information easier to digest.
|
|
||
| GA APIs should match the exact set of APIs enabled in GA for the Kubernetes version | ||
| the emulation version is set to. | ||
| Instead, we are proposing to keep forward compatibility of all APIs in compatibility version mode: |
There was a problem hiding this comment.
We're proposing that non-emulatable feature controllers to update to use the newest available API, ignoring emulation API enablement, right? Maybe say this more directly?
There was a problem hiding this comment.
Yes, mainly for controllers for now. May expand to kublet if we want to add compat version in node. Updated the section.
| So this exception should only be used if the feature | ||
| 1. is Beta and disabled by default for at least one version in the range of `1.{binaryMinorVersion-3}..1.{binaryMinorVersion-1}`. | ||
| 1. is going through a lot of changes when it is off by default, and the feature implementation history is very hard to preserve. | ||
| 1. has stabilized after it is enabled by default. |
There was a problem hiding this comment.
Are un-emulatable features intended as a stop-gap during beta or is this intended to be a permanent property of the emulation versions feature?
If this is Beta only, let's add a line item in the graduation criteria about resolving this before GA.
There was a problem hiding this comment.
My intention is to have it as a permanent property of the emulation versions feature. The API forward compatibility is a workaround for non-emulatable controller features. But there could be really difficult cases where non of the options work. We should have something to handle that instead of letting the feature be emulated with unpredictable results.
siyuanfoundation
force-pushed
the
compat-versions-api
branch
2 times, most recently
from
48e2f87 to
f8bce11
Compare
…Features. Signed-off-by: Siyuan Zhang <[email protected]>
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
f8bce11 to
961b0c9
Compare
siyuanfoundation
changed the title
