remove usage of flexvolume in kubeadm

[neolit123]

distroless effort:
kubernetes/kubernetes#70249

see this message from sig-storage https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/kubernetes-dev/zC8jCLg142w/2P3BN5oTAgAJ

We are gauging the viability of deprecating the Flexvolume master API calls.
...
There is an effort underway to move to core k8s component images to distroless.

kube-apiserver and kube-scheduler already moved to distroless, but the kube-controller-manager was blocked due to flexvolume. kube-proxy is yet to move too.

currently kubeadm has related logic to manage flex volumes for the kube-controller-manager static Pod:

  - hostPath:
      path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      type: DirectoryOrCreate
    name: flexvolume-dir

    - mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
      name: flexvolume-dir

IIRC the above is currently GA and required for the KCM to run properly.
in case the KCM / kubelet deprecates and remove flexvoluime support kubeadm should follow.

upstream ticket:
kubernetes/kubernetes#98815

[xlgao-zju]

@neolit123 Since kubernetes/kubernetes#91329 has been merged, shall we remove the mount of flexvolume-dir now?

[neolit123]

so my understanding is that flex volume support does not work with the distroless image for the KCM that k8s ships by default now. however, someone might decide to build their own image and override the one that kubeadm uses from k8s.gcr.io.

this leads me to believe that instead of removing the kubeadm support for flex volume today, we should wait until flex volume is completely removed.

cc @dims please correct me if needed on this one.
cc @rosti

[dims]

yep well said @neolit123

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[neolit123]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[BenTheElder]

kube-proxy distroless would require us to do the iptables bits so I'm not sure if anyone will pick that up but is maybe still a good idea.

flex volume seems problematic in general, because e.g. updating the userspace in this distroful images is also technically perhaps a breaking change...

But it still needs an official timeline.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fabriziopandini]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[fabriziopandini]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[neolit123]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[neolit123]

/remove-lifecycle stale

[BenTheElder]

Seems like we should just freeze this until there's a pending timeline for removal from kubernetes that kubeadm can follow?

[neolit123]

Yep, that is true.
/lifecycle frozen

[pacoxu]

Deprecation of FlexVolume
FlexVolume is deprecated. Out-of-tree CSI driver is the recommended way to write volume drivers in Kubernetes. See this doc for more information. Maintainers of FlexVolume drivers should implement a CSI driver and move users of FlexVolume to CSI. Users of FlexVolume should move their workloads to CSI driver.

FlexVolume was deprecated since v1.23.

But according to https://github.com/kubernetes/community/blob/master/sig-storage/volume-plugin-faq.md#working-with-out-of-tree-volume-plugin-options

So you can continue to use it without worry of deprecation, but note that additional features (like topology, snapshots, etc.) will only be added to CSI not to FlexVolume.

FlexVolume was deprecated but will not be removed, but will be still maintained.(No new feature.)

[neolit123]

we can remove the kubeadm integration which will be fairly easy, but perhaps we should do it once k/k core removes it. if we remove it now it's not clear how many kubeadm users we will break.

[pacoxu]

Upgrade behavior should not be changed. So if user upgrades their cluster, the cluster will not break.
Can we just change kubeadm init to not use flex volume? Or just keep it as is.

[neolit123]

i think after kubeadm upgrade if the manifests no longer have the flex volume bits, an existing user of the deprecated flex volume support from core k8s will be broken.

for init it can be done, but we should tie it to the k8s core removal instead.