From 961b0c9334603734f9b8e074430a349f75f630bd Mon Sep 17 00:00:00 2001 From: Siyuan Zhang Date: Mon, 13 Jan 2025 16:52:04 -0800 Subject: [PATCH] Change API availability to forward compatibility and add UnEmulatableFeatures. Signed-off-by: Siyuan Zhang --- .../4330-compatibility-versions/README.md | 91 ++++++++++++++++--- 1 file changed, 77 insertions(+), 14 deletions(-) diff --git a/keps/sig-architecture/4330-compatibility-versions/README.md b/keps/sig-architecture/4330-compatibility-versions/README.md index 3541dc2a274..783ca6b413e 100644 --- a/keps/sig-architecture/4330-compatibility-versions/README.md +++ b/keps/sig-architecture/4330-compatibility-versions/README.md @@ -90,10 +90,12 @@ tags, and then generate with `hack/update-toc.sh`. - [Changes to Feature Gates](#changes-to-feature-gates) - [Feature Gate Lifecycles](#feature-gate-lifecycles) - [Feature gating changes](#feature-gating-changes) + - [Non-Emulatable Features](#non-emulatable-features) - [Validation ratcheting](#validation-ratcheting) - [CEL Environment Compatibility Versioning](#cel-environment-compatibility-versioning) - [StorageVersion Compatibility Versioning](#storageversion-compatibility-versioning) - [API availability](#api-availability) + - [Alternatives to API forward compatibility](#alternatives-to-api-forward-compatibility) - [API Field availability](#api-field-availability) - [Discovery](#discovery) - [Version introspection](#version-introspection) @@ -562,6 +564,19 @@ func ClientFunction() { ``` +#### Non-Emulatable Features +In very rare cases, if the feature implementation history could not be preserved in the code base with reasonable maintenance cost, a feature could opt out of the compatibility version through the `NonEmulatableFeatures` list. + +With the `NonEmulatableFeatures` list, the server would fail to start if +1. the `EmulationVersion != BinaryVersion` and +1. any feature in the `NonEmulatableFeatures` list is disabled by default at the emulation version and explicitly enabled by the `--feature-gates` flag. + +So this exception should only be used if the feature +1. is Beta and disabled by default for at least one version in the range of `1.{binaryMinorVersion-3}..1.{binaryMinorVersion-1}`. +1. is going through a lot of changes when it is off by default, and the feature implementation history is very hard to preserve. +1. has stabilized after it is enabled by default. +1. Removing the feature out of `NonEmulatableFeatures` should be part of the feature's GA criteria to ensure the list would not just keep growing. + ### Validation ratcheting All validationg ratcheting needs to account for compatibility version. @@ -610,20 +625,68 @@ The storage version of each group-resource is the newest ### API availability -Similar to feature flags, all APIs group-versions declarations will be modified -to track which Kubernetes version the API group-versions are introduced (or -removed) at. +Ideally, similar to feature flags, all API's group-version-resource declarations should be modified +to track which Kubernetes version the GVRs are introduced (or +removed) at: if an API is introduced after (or removed before) the emulation version, it should not be available at the emulation version. + +But in practice, that would make the controller code changes intractable if an API is graduating from Beta to GA and the controller wants to use newer API. For example, to graduate Multiple Service CIDRs to GA, normally the controller code change would look like: +```diff +- c.serviceCIDRInformer = networkingv1beta1informers.NewFilteredServiceCIDRInformer(client, 12*time.Hour, ++ c.serviceCIDRInformer = networkingv1informers.NewFilteredServiceCIDRInformer(client, 12*time.Hour, + cache.Indexers{cache.NamespaceIndex: cache.MetaNamespaceIndexFunc}, + func(options *metav1.ListOptions) { + options.FieldSelector = fields.OneTermEqualSelector("metadata.name", DefaultServiceCIDRName).String() + }) + +- c.serviceCIDRLister = networkingv1beta1listers.NewServiceCIDRLister(c.serviceCIDRInformer.GetIndexer()) ++ c.serviceCIDRLister = networkingv1listers.NewServiceCIDRLister(c.serviceCIDRInformer.GetIndexer()) + c.serviceCIDRsSynced = c.serviceCIDRInformer.HasSynced + + return c + //... -GA APIs should match the exact set of APIs enabled in GA for the Kubernetes version -the emulation version is set to. +type Controller struct { + eventRecorder record.EventRecorder -All Beta APIs (both off-by-default and on-by-default, if any of those -still exist) need to behave exactly as they did for the Kubernetes version -the emulation version is set to. I.e. `--runtime-config=api/` needs -to be able to turn on APIs exactly like it did for each Kubernetes version that -emulation version can be set to. + serviceCIDRInformer cache.SharedIndexInformer +- serviceCIDRLister networkingv1beta1listers.ServiceCIDRLister ++ serviceCIDRLister networkingv1listers.ServiceCIDRLister + serviceCIDRsSynced cache.InformerSynced -Alpha APIs may not be enabled in conjunction with emulation version. +``` +To fully emulate the controller for an older version, anywhere v1 api/type is referenced, it would need to switch to the v1beta version if the emulation version is older than the binary version. This would mean a lot of extra work, complicated testing rules, and high maintenance cost even for simple API graduations, while the emulation fidelity is unreliable with the extra complexity. + +So instead of truly emulating the feature controllers and API availability at the emulation version, we are proposing to keep forward compatibility of all APIs in compatibility version mode and have the non-emulatable feature controllers to update to use the newest available API. + +For API availability, this means: +1. if an API is removed (as indicated by the GVK prerelease lifecycle) after the emulation version, it would still be available at the emulation version. +1. if an API is Beta at the emulation version (meaning the Beta API has been introduced and has not been removed by the emulation version), it can be enabled by `--runtime-config=api/` or it can be on-by-default at the emulation version. If a Beta API is enabled at the emulation version, and it has GAed between the emulation version and the binary version, its GA version would also be enabled at the emulation version. if a newer Beta API is introduced between the emulation version and the binary version, the newer Beta API would also be enabled at the emulation version. +1. If an API has GAed and has not been removed at the emulated version, it would be enabled by default at the emulation version. If a newer stable version of the GA API has been introduced between the emulation version and the binary version, the new GA API would also be enabled at the emulation version along with the old GA API. +1. Alpha APIs may not be enabled in conjunction with emulation version. + +Here are some examples for `BinaryVersion = 1.33`: +API Prerelease Lifecycle | EmulationVersion | APIs Available @EmulationVersion +-----|-----|----- +`v1alpha1: introduced=1.30, removed=1.31`
`v1beta1: introduced=1.31, removed=1.32`
`v1: introduced=1.32` | 1.30 | NA +`v1alpha1: introduced=1.30, removed=1.31`
`v1beta1: introduced=1.31, removed=1.32`
`v1: introduced=1.32` | 1.31 | `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true` +`v1alpha1: introduced=1.30, removed=1.31`
`v1beta1: introduced=1.31, removed=1.32`
`v1: introduced=1.33` | 1.33 | `api/v1` +`v1beta1: introduced=1.31, removed=1.32`
`v1beta2: introduced=1.32` | 1.31 | `api/v1beta1` and `api/v1beta2` available only when `--runtime-config=api/v1beta1=true` +`v1beta1: introduced=1.31, removed=1.32`
`v1beta2: introduced=1.32` | 1.33 | `api/v1beta2` available only when `--runtime-config=api/v1beta2=true` +`v1: introduced=1.28, removed=1.32`
`v2beta1: introduced=1.31, removed=1.32`
`v2: introduced=1.32` | 1.30 | `api/v1` +`v1: introduced=1.28, removed=1.32`
`v2beta1: introduced=1.31, removed=1.32`
`v2: introduced=1.32` | 1.31 | `api/v1` always available, `api/v2beta1` and `api/v2` available only when `--runtime-config=api/v2beta1=true` +`v1: introduced=1.28, removed=1.32`
`v2beta1: introduced=1.31, removed=1.32`
`v2: introduced=1.32` | 1.33 | `api/v2` + +For the controller, at the emulation version the controller is still enabled by enabling the Beta API **AND** the controller feature as before, but under the hood the controller is calling the newer API. + +The forward compatibility of API availability should not affect data compatibility because storage version is still controlled by the `MinCompatibilityVersion` regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is `Equivalent`. + +#### Alternatives to API forward compatibility +To make API graduation workable for controller code change under compatibility version, we have considered and rejected the following alternative options: +1. `if .. else ..` statements everywhere is just impractical. +1. have `v1Controller` and `v1beta1Controller` code in separate files would mean duplicate maintenance, test work, and developer churn. +1. have some smart wrapper to pick the right version for each API/type reference: it is very hard to design a generic enough wrapper for all cases. +1. convert the data to older version when the newer API is called: this is the essentially same as the enabling the newer API. +1. have special mechanisms for controllers in `k/k` to call v1 apis, but not expose the v1 apis to clients: clients can spend efforts to duplicate the special mechanisms. ### API Field availability @@ -831,9 +894,9 @@ Alpha feature graduated to Beta|off-by-default|on-by-default|feature enabled onl Beta feature graduated to GA|on-by-default|on|feature enabled unless `--feature-gates==false`|feature always enabled, feature gate may not be set Beta feature removed|on-by-default|-|feature enabled unless `--feature-gates==false`|feature does not exist Alpha API introduced|-|off-by-default|API does not exist|alpha APIs may not be used in conjunction with emulation version -Beta API graduated to GA|off-by-default|on|API available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available -Beta API removed|off-by-default|-|API available only when `--runtime-config=api/v1beta1=true`|API `api/v1beta1` does not exist -on-by-default Beta API removed|on-by-default|-|API available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist +Beta API graduated to GA|off-by-default|on|API `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available +Beta API removed|off-by-default|-|API `api/v1beta1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1beta1` does not exist +on-by-default Beta API removed|on-by-default|-|API `api/v1beta1` available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist API Storage version changed|v1beta1|v1|Resources stored as v1beta1|Resources stored as v1 new CEL function|-|function in StoredExpressions CEL environment|CEL function does not exist|Resources already containing CEL expression can be evaluated introduced CEL function|function in StoredExpressions CEL environment|function in NewExpressions CEL environment|Resources already containing CEL expression can be evaluated|CEL expression can be written to resources and can be evaluted from storage