Argo Workflows install via the helm chart and managed namespaces

[SleepyBrett]

I went through an exercise yesterday trying to get Workflows installed using the managed namespace feature: https://argoproj.github.io/argo-workflows/managed-namespace/ and I ran into a few problems, and I don't see a lot of documented guidance around how to work around these problems. So I thought I'd present them here and figure out what issues should be cut or not cut to alleviate this for those that come after me.

So my requirements:

• Run argo-workflows controller/server in the namespace argo
• Only process workflows submitted the workflows namespace and keep the workflow pods in that namespace as well
• Use the helm chart to install. (using 0.20.0)

My values:

    singleNamespace: true
    controller:
      workflowNamespaces:
        - argo-workflows
      extraArgs:
        - --managed-namespace
        - workflows

Helm command passed -n argo.

Quick note, I'm not fully versed in the controller, but is there a situation where there are workflowNamespaces that are not managed-namespaces?. I only ask because after a check of the helm chart it was clear that I needed to use .Values.controller.extraArgs to add the --managed-namespace flag.

Initial Result:
The workflows controller and server properly spin up in the argo namespace, but shortly thereafter the controller pod went into crashloop. Logs indicate that the argo-controller service account lacked permissions in the workflows namespace. I confirmed that the workflows namespace contained no rolebindings that would allow the argo-controller service account any permissions in the target managed namespace.

I then turned to the documentation briefly and found no mention of what permissions that the controller needed in order to do it's job. I then turned to the helm chart. I started by looking here: https://github.com/argoproj/argo-helm/blob/argo-workflows-0.20.0/charts/argo-workflows/templates/controller/workflow-role.yaml#L2 .. and I gotta say, I've written my share of helm charts with complicated statements, but that one is really hard to parse. However I assume the intent here is "create a list of all the workflowNamespaces both the new value and the deprecated value and the release namespace, then use that list to create this role in every namespace". However that did not seem to happen, I only got one copy of this role. This may be a helm thing, last I checked helm really really hates to create things outside of the release namespace.

Hacks to make it work:
In the end I had to create a few things in the workflow namespace:

• I created a service account to run my workflow pods workflow-runner
• I bound that role to the admin clusterrole to give it namespace admin rights as a catchall, I'd love to reduce this rbac down to something more reasonable though.
• I also bound the workflow controller sa from the argo namespace to admin in the namespace as an initial stab, but that was apparently not enough.
• I eventually created the role below and bound it to the controller service account to quiet all the other errors the controller was spewing. (in addition to the already bound admin clusterrole).

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  labels:
  name: argo-workflows-additional
  namespace: workflows
rules:
  - apiGroups:
    - ""
    resources:
    - events
    verbs:
    - create
    - update
    - patch
    - get
    - list
    - watch
    - delete
  - apiGroups:
    - argoproj.io
    resources:
    - workflows
    - workflows/finalizers
    - workflowtasksets
    - workflowtasksets/finalizers
    - workflowartifactgctasks
    verbs:
    - get
    - list
    - watch
    - update
    - patch
    - delete
    - create
  - apiGroups:
    - argoproj.io
    resources:
    - workflowtaskresults
    - workflowtaskresults/finalizers
    verbs:
    - list
    - watch
    - deletecollection

Once those hacks were in place things became healthy.

Questions:

• Is this a use-case that the helm-chart was intended to address and did I simply miss something in the values that would have magically made it work without my hacks?
• If not, a small guide on this page would help: https://argoproj.github.io/argo-workflows/managed-namespace/
• It would be nice if that guide also contained the rbac role that needs to be bound to the controller service account in the managed namespaces.

[vitalyrychkov]

Just went almost the same way testing managed-namespace with Argo v3.4.7. Finally I gave up on Helm and downloaded the manifest file to play around.

In the "managed" namespace I have created a copy of Roles and Bindings, bound to argo user in the argo namespace, but it was not possible to run any workflow without creating a ClusterRole/Binding for the argo user to access ClusterWorkflowTemplates. In my opinion, this ClusterRole requirement undermines the idea of the namespaced (non-cluster) Argo deployment.

The main issue does not depend on the deployment way, e.g. Helm/manifest: we need some clear idea what is a "managed-namespace deployment". My understanding is that managed-namespace deployment must not require Cluster-scoped Roles. There must be two sets of Roles/Bindings for the argo namespace and for the managed namespace(s). I would consider creating the issue in the Argo Workflows GitHub.

I am not good in writing charts, therefore no suggestions how to change them.