[WIP] ✨ Add clusterctl changes for cross-ns CC ref #11395

Danil-Grigorev · 2024-11-11T10:19:06Z

What this PR does / why we need it:
This PR allows clusterctl to create a template with a reference to CC located in a different namespace.
This change allows to perform e2e tests in a multi-ns environment.

Added e2e tests to verify cluster-class quick-start scenario and runtime extension scenario in a cross-namespaced context.

The PR is built on top of:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5673

k8s-ci-robot · 2024-11-11T10:19:15Z

This PR is currently missing an area label, which is used to identify the modified component when generating release notes.

Area labels can be added by org members by writing /area ${COMPONENT} in a comment

Please see the labels list for possible areas.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

fabriziopandini · 2024-11-25T14:02:50Z

Renamed wip since it builds on a PR not yet merged

fabriziopandini

Made first pass, I will get back to this when the PR it builds on top are merged

fabriziopandini · 2024-11-25T14:15:39Z

test/e2e/data/infrastructure-docker/main/cluster-template-cross-ns-topology/cluster.yaml

+  name: '${CLUSTER_NAME}'
+spec:
+  topology:
+    classNamespace: '${CLUSTER_CLASS_NAMESPACE}'


Wondering if we can avoid creating a new template by changing cluster-with-topology.yaml so it uses empty string as a default value in case CLUSTER_CLASS_NAMESPACE is not set

fabriziopandini · 2024-11-25T14:19:28Z

test/e2e/cross-ns-quick-start.go

+// This test is meant to provide a first, fast signal to detect regression; it is recommended to use it as a PR blocker test.
+// NOTE: This test works with Clusters with and without ClusterClass.


Considering we are trying to keep PR-blocking duration as short as possible, and that the code paths touched by this change are limited and very specific to CC management, would it be possible to add this test on a PR only on demand + on periodic (once it is added to a PR, then it became blocking)

Currently using this as a way to quickly test subset of scenarios without having to invoke full-test-suite. I’ll unmark it from pre-submits. Should all added test cases be on demand + perodic?

fabriziopandini · 2024-11-25T14:26:12Z

test/e2e/cross-ns-quick-start.go

+		namespace, cancelWatches = framework.SetupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, input.PostNamespaceCreated)
+		clusterNamespace, cancelWatchesCluster = framework.SetupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, input.PostNamespaceCreated)


What about having the cluster in the spec namespace (as we usually do) and create a second namespace for the CC with a name equal to the first namespace + a "cluster-class" suffix; also we probably don't need watches in the CC namespace.

This way when we look at the test, most of the triage work will be consistent with what we usually do, and in case we need to look at the CC, it will be easy to find as it is because it is a namespace with the same root name as everything else

fabriziopandini · 2024-11-25T14:45:26Z

cmd/clusterctl/client/clusterclass.go

@@ -44,7 +46,7 @@ func addClusterClassIfMissing(ctx context.Context, template Template, clusterCla
 		return template, nil
 	}

-	clusterClassesTemplate, err := fetchMissingClusterClassTemplates(ctx, clusterClassClient, clusterClient, classes, targetNamespace, listVariablesOnly)
+	clusterClassesTemplate, err := fetchMissingClusterClassTemplates(ctx, clusterClassClient, clusterClient, classes, listVariablesOnly)


Just to check if I'm getting this right.
As of today, when we are deploying objects, clusterctl allows to override the namespace set into templates (use case: the user wants to deploy to namespace xyz instead of namespace original).

With this PR we are changing this behaviour, and always preserving the namespace from the template when we are adding the cluster class to the template being deployed (cc preserve: namespace original).

If this is correct, and if I'm not wrong, this change might break users that are deploying a Cluster with ClassNamespace empty, and are expecting that also the CC gets deployed in the target namespace xyz along side with the cluster.

It shouldn’t break these users, since regular process of overriding the namespace of the templated resources is still performed. This new field was never a part of this logic, and the SoT here is the template initially.

This change avoids problems in discovering the existence of the CC in the specified namespace, allowing to template only a Cluster resource.

There is no support in clusterclt to distinguish that some of the resources should go into A namespace, and some other into B, so this multi-namespace setup is intended to happen in a multiple passes.

ClusterClass and referenced templates

A cluster resource in a different namespace

repeat 2 for any namespace without failures on clusterctl side, or any existing template modifications.

I'm not entirely sure, but I think this shouldn't break anything.

The namespace-thing for cluster-templates is only envsubst, right?

(while for provider installations it's a lot more)

EDIT: I'll take that one back

My best guess is that it still works as good as before :)

fabriziopandini · 2024-11-25T14:48:15Z

cmd/clusterctl/client/clusterclass_test.go

+			clusterInitialized:          false,
+			objs:                        []client.Object{},
+			targetNamespace:             "ns1",
+			clusterClassNamespace:       "ns2",


Can we have a similar test when both namespaces are the same, and one where both are empty?

k8s-ci-robot · 2025-01-07T19:57:36Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sbueringer for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sbueringer

Initial round of review.

But let's wait a bit with adjustments. I have to talk to @fabriziopandini a bit about the test coverage we want / need

sbueringer · 2025-01-14T17:53:23Z

test/e2e/cluster_upgrade_runtimesdk.go

-		namespace     *corev1.Namespace
-		cancelWatches context.CancelFunc
+		input                             ClusterUpgradeWithRuntimeSDKSpecInput
+		namespace, infraNamespace         *corev1.Namespace


Maybe let's also call this clusterClassNamespace
(and cancelClusterClassNamespaceWatches below)

sbueringer · 2025-01-14T17:57:34Z

test/e2e/cluster_upgrade_runtimesdk.go

@@ -95,6 +95,9 @@ type ClusterUpgradeWithRuntimeSDKSpecInput struct {
 	ExtensionServiceNamespace string
 	// ExtensionServiceName is the name of the service to configure in the test-namespace scoped ExtensionConfig.
 	ExtensionServiceName string
+
+	// ClassNamespace is an optional class namespace reference, configuring cross-namespace cluster class reference


the godoc sounds a bit strange and also the field name :)

Is this more like: DeployClusterClassInSeparateNamespace ?

Hm just noticed that it is a *string in the quickstart test. Do we care about the specific namespace used or is a boolean flag enough?
(would be good if both tests have ~ the same input parameter)

I started with bool, but then realized that the NS name generation has a seed, specName in this case. And these names may overlap, so I left the value being specified on demand. May be also clear when collecting logs, where the resources are located (not just 2 randomly generated names as it would otherwise be)

sbueringer · 2025-01-14T17:59:29Z

test/e2e/quick_start.go

+			variables["CLUSTER_CLASS_NAMESPACE"] = *input.ClassNamespace
+		}
+
+		By("Creating a cluster referencing a clusterClass from another namespace")


Not always true

- Add documentation on securing cross-namespace access for CC - Add ByClusterClassRef index - Support cross-ns CC rebase Signed-off-by: Danil-Grigorev <[email protected]>

Signed-off-by: Danil-Grigorev <[email protected]>

k8s-ci-robot · 2025-01-15T18:36:02Z

@Danil-Grigorev: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-e2e-blocking-main	`0ee7554`	link	true	`/test pull-cluster-api-e2e-blocking-main`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-area PR is missing an area label labels Nov 11, 2024

k8s-ci-robot requested a review from elmiko November 11, 2024 10:19

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Nov 11, 2024

k8s-ci-robot requested a review from jackfrancis November 11, 2024 10:19

Danil-Grigorev force-pushed the standardize-references-full branch 5 times, most recently from d92a5c2 to 7560e66 Compare November 12, 2024 11:33

Danil-Grigorev mentioned this pull request Nov 12, 2024

🐛 Use namespace of the reference on external.Get #11361

Merged

Danil-Grigorev changed the title ~~[WIP] ✨ E2E test for the classNamespace usage~~ [WIP] ✨ Add clusterctl changes for cross-ns CC ref Nov 12, 2024

Danil-Grigorev changed the title ~~[WIP] ✨ Add clusterctl changes for cross-ns CC ref~~ ✨ Add clusterctl changes for cross-ns CC ref Nov 12, 2024

k8s-ci-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Nov 12, 2024

Danil-Grigorev force-pushed the standardize-references-full branch from 7560e66 to 09ce978 Compare November 18, 2024 19:46

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 18, 2024

fabriziopandini mentioned this pull request Nov 21, 2024

Enabling option to ignore namespaces #11193

Open

fabriziopandini changed the title ~~✨ Add clusterctl changes for cross-ns CC ref~~ [WIP] ✨ Add clusterctl changes for cross-ns CC ref Nov 25, 2024

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 25, 2024

fabriziopandini reviewed Nov 25, 2024

View reviewed changes

Danil-Grigorev force-pushed the standardize-references-full branch from 09ce978 to 2d4a64e Compare December 4, 2024 14:34

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 4, 2024

Danil-Grigorev force-pushed the standardize-references-full branch from 2d4a64e to c181e3e Compare December 4, 2024 15:22

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 4, 2024

Danil-Grigorev force-pushed the standardize-references-full branch from c181e3e to e155375 Compare December 5, 2024 12:18

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 5, 2024

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 21, 2024

Danil-Grigorev force-pushed the standardize-references-full branch from e155375 to 9b2cce5 Compare January 7, 2025 19:57

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 7, 2025

Danil-Grigorev mentioned this pull request Jan 10, 2025

✨ Add classNamespace to topology #11352

Open

Danil-Grigorev force-pushed the standardize-references-full branch from 9b2cce5 to b2cbee3 Compare January 14, 2025 17:37

sbueringer reviewed Jan 14, 2025

View reviewed changes

Danil-Grigorev added 5 commits January 15, 2025 17:01

Add classNamespace to topology

ebdc21d

- Add documentation on securing cross-namespace access for CC - Add ByClusterClassRef index - Support cross-ns CC rebase Signed-off-by: Danil-Grigorev <[email protected]>

Add e2e test to cover crossNamespace provisionging

a3549fb

Signed-off-by: Danil-Grigorev <[email protected]>

Prevent duplicate CC creation in template processing

de7239a

Signed-off-by: Danil-Grigorev <[email protected]>

Test runtimextension integration

580b0ad

Signed-off-by: Danil-Grigorev <[email protected]>

Refactor to use empty CC namespace, add tests

dc07405

Signed-off-by: Danil-Grigorev <[email protected]>

Danil-Grigorev force-pushed the standardize-references-full branch 2 times, most recently from 3084938 to 9b17207 Compare January 15, 2025 17:09

Add changes to CC rebase e2e test

0ee7554

Signed-off-by: Danil-Grigorev <[email protected]>

Danil-Grigorev force-pushed the standardize-references-full branch from 9b17207 to 0ee7554 Compare January 15, 2025 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] ✨ Add clusterctl changes for cross-ns CC ref #11395

[WIP] ✨ Add clusterctl changes for cross-ns CC ref #11395

Danil-Grigorev commented Nov 11, 2024 •

edited

Loading

k8s-ci-robot commented Nov 11, 2024

fabriziopandini commented Nov 25, 2024

fabriziopandini left a comment

fabriziopandini Nov 25, 2024

fabriziopandini Nov 25, 2024

Danil-Grigorev Jan 15, 2025

fabriziopandini Nov 25, 2024

fabriziopandini Nov 25, 2024

Danil-Grigorev Dec 4, 2024

sbueringer Jan 14, 2025 •

edited

Loading

sbueringer Jan 14, 2025 •

edited

Loading

fabriziopandini Nov 25, 2024

k8s-ci-robot commented Jan 7, 2025

sbueringer left a comment

sbueringer Jan 14, 2025

sbueringer Jan 14, 2025

Danil-Grigorev Jan 15, 2025

sbueringer Jan 14, 2025

k8s-ci-robot commented Jan 15, 2025

		// This test is meant to provide a first, fast signal to detect regression; it is recommended to use it as a PR blocker test.
		// NOTE: This test works with Clusters with and without ClusterClass.

		namespace, cancelWatches = framework.SetupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, input.PostNamespaceCreated)
		clusterNamespace, cancelWatchesCluster = framework.SetupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, input.PostNamespaceCreated)

[WIP] ✨ Add clusterctl changes for cross-ns CC ref #11395

Are you sure you want to change the base?

[WIP] ✨ Add clusterctl changes for cross-ns CC ref #11395

Conversation

Danil-Grigorev commented Nov 11, 2024 • edited Loading

k8s-ci-robot commented Nov 11, 2024

fabriziopandini commented Nov 25, 2024

fabriziopandini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbueringer Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

sbueringer Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 7, 2025

sbueringer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Jan 15, 2025

Danil-Grigorev commented Nov 11, 2024 •

edited

Loading

sbueringer Jan 14, 2025 •

edited

Loading

sbueringer Jan 14, 2025 •

edited

Loading