Evaluation of reclass and possible inventory alternatives #1007

MatteoVoges · 2023-05-10T09:33:17Z

MatteoVoges
May 10, 2023

Hello, I want to share my thoughts about the current inventory solution with reclass, pro's and con's of it and other inventory alternatives and the introduction of a pluggable inventory system.

Current Solution with reclass

Advantages ✔️

good structure (targets and classes) by default
meta information for targets

Disadvantages ❌

old code (legacy python2) --> hard to extend and maintain
currently no support for default values and interpolation with relative paths (but upcoming)
no python package, reclass as submodule isn't that nice

Your opinion

Now your opinion and ideas are needed:
What do you like about reclass? What features are missing to improve your workflows? What else could be added to my list above?

Inventory alternatives

OmegaConf

Advantages ✔️

is actively maintained, regularly released
many users (if it's used, then it has to be good )
support of relative paths and default values
support of custom resolvers --> opportunity to define our own tags
easy to extend due open-source and possible collaborations
python package

Disadvantages ❌

no structure by default, but easy to adapt the structure of reclass, or think of a new structure!
some mismatches in syntax to reclass, but easy to fix with a migration-script (no disadvantage at all)

There are several more options, but OmegaConf seems really easy to use, easy to integrate into kapitan and its actively maintained.

All type of feedback is welcome, so ask questions, share your thoughts with us and let us also know, which features you want to have in your inventory in general!

Other

figgy: Cloud bases config with secret management and own version control
- not suitable for kapitan as we need local yaml config
yamlpath: yaml library that supports loading and merging nodes, but no interpolation
hiyapyco: yaml config
- supports loading and merging nodes
- interpolation only with jinja2
- uses GNU-3.0 License

References

ademariag · 2023-05-17T09:15:01Z

ademariag
May 17, 2023
Maintainer

Thank you @MatteoVoges, great work. Is there a branch with a working setup that we could test?

1 reply

MatteoVoges May 17, 2023
Author

I'm currently working on a migration script that turns reclass inventories into omegaConf compatible inventories. Then I could apply this to our example inventory and publish a finished version on a branch in the kapitan repo, so that you don't have to checkout the nexenio fork. The core features of the inventory are already available, but some flags (like compose-node-name and such) have to be done.

MatteoVoges · 2023-06-16T14:15:10Z

MatteoVoges
Jun 16, 2023
Author

Is there a branch with a working setup that we could test?

You can test now the new features regarding omegaconf:

Installation and setup

Docker

Create a new Dockerfile and paste the following contents in there:

Dockerfile:

FROM python:3.10-slim AS python-builder

USER root

RUN apt-get update \
    && apt-get install --no-install-recommends -y \
    curl \
    build-essential \
    git \
    ssh-client \
    libmagic1 \
    gnupg \
    ca-certificates \
    gosu

RUN git clone \
    --recurse-submodules -j8 \
    --branch pluggable-inventory \
    https://github.com/neXenio/kapitan.git /kapitan

WORKDIR /kapitan

# build kapitan
RUN pip install -e .

# Install Helm
RUN curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
  && chmod 700 get_helm.sh \
  && HELM_INSTALL_DIR=/usr/local/bin ./get_helm.sh \
  && rm get_helm.sh

RUN apt-get install -y default-jre

# install omegaconf
RUN make pull_oc

ENTRYPOINT ["kapitan"]

Build the image with docker build <path> -t kapitan:dev. Now replace your image in your ./kapitan script.

Manual (Dev Mode)

Create a new virtual enviroment: python -m venv omegaconfenv
activate your environment source omegaconf/bin/activate
Install kapitan

git clone --branch pluggable-inventory https://github.com/nexenio/kapitan.git
cd kapitan
git submodule update --init
poetry install

Install Omegaconf: make pull_oc

Usage

First I highly recommend you to backup your inventory or creating a test inventory, in case something works not as expected.
To activate the feature you have to enable the flag: --omegaconf. When you run this the first time you also have to specify --migrate, to replace the yaml paths in the references.

So run kapitan compile --omegaconf --migrate. The inventory speed could be slower than it was before with reclass, but there are lots of performance boosters planned.

If you installed the project in dev mode, you are able to experiment with costum resolvers. At the moment there are several resolvers already enabled. Here are some highlights:

${env:PATH}: access environment variables
${select:<yaml_path>, default} : interpolation with default value, if the path doesn't exist
${key:}: get the name of the key
${fullkey:}: get the full name of the key

You can extend these resolvers or implement your own in kapitan/omegaconf_inv.py . Starting at line 148 you can modify the existing resolver dep and define any python object you can imagine. Note that you can input several input arguments and treat a resolver as a regular python function.

If you have problems with setup or questions in general, please leave them here and I'm happy to answer them.

0 replies

uberspot · 2023-06-21T16:22:39Z

uberspot
Jun 21, 2023
Maintainer

How much effort would it be to port reclass to python3 and compile it as a package do you think? VS migrating everyone's inventories and semantics. Cause even with OmegaConf it seems like you'll still need a translation later from inventory yaml to omegaconf constructs?

1 reply

MatteoVoges Jun 29, 2023
Author

The effort for migration and the difference in syntax will be as small as possible. At the moment the only thing is, that the paths in the interpolations will use . instead of :, because they are the official yaml-path-notation. This happens in a migration step, so you don't have to touch thousands of files.
The main benefit from using omegaconf is targetwise-loading (future speedup potential) and the use of custom resolvers. They work like a generator for the inventory and can create large configurations with very less specification.
I will be presenting my results at the workgroup-meetings, so please join, if you want to see some examples or if you have more questions.

ademariag · 2023-06-27T15:51:13Z

ademariag
Jun 27, 2023
Maintainer

@uberspot the reasons are not just to do with python3 port, but in general to do with the inability to easily extend and improve on reclass. Plan is to maintain compatibilty with reclass, so this would be an addition for people that want to move to use it.

0 replies

jmmclean · 2025-01-09T15:40:32Z

jmmclean
Jan 9, 2025

I saw that omegaconf support was added in #1173, however the migration functionality does not work. Any support would be appreciated!

> kapitan compile --inventory-backend omegaconf --migrate
class file not found for class common.global, tried ['./inventory/classes/common/global/init.yml', './inventory/classes/common/global.yml', './inventory/classes.yml', './inventory/classes/init.yml']
aws-us-west-2-dca-wl-prd-001: could not render due to error Class common.global not found
class file not found for class common.global, tried ['./inventory/classes/common/global/init.yml', './inventory/classes/common/global.yml', './inventory/classes.yml', './inventory/classes/init.yml']
azr-westus-dca-wl-prd-001: could not render due to error Class common.global not found
class file not found for class common.global, tried ['./inventory/classes/common/global/init.yml', './inventory/classes/common/global.yml', './inventory/classes.yml', './inventory/classes/init.yml']
class file not found for class common.global, tried ['./inventory/classes/common/global/init.yml', './inventory/classes/common/global.yml', './inventory/classes.yml', './inventory/classes/init.yml']
aws-us-east-1-dca-wl-lab-002: could not render due to error Class common.global not found
aws-us-west-2-dca-wl-prd-002: could not render due to error Class common.global not found
Error while loading the OmegaConf inventory
Traceback (most recent call last):
  File "/opt/venv/bin/kapitan", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kapitan/cli.py", line 639, in main
    args.func(args)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/cli.py", line 76, in trigger_compile
    compile_targets(
  File "/opt/venv/lib/python3.11/site-packages/kapitan/targets.py", line 42, in compile_targets
    inventory = get_inventory(inventory_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kapitan/resources.py", line 345, in get_inventory
    inventory_backend = backend(
                        ^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/backends/omegaconf/__init__.py", line 69, in __init__
    super().__init__(*args, **kwargs, target_class=OmegaConfTarget)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/inventory.py", line 51, in __init__
    self.__initialise(ignore_class_not_found=ignore_class_not_found)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/inventory.py", line 92, in __initialise
    self.render_targets(self.targets, ignore_class_not_found=ignore_class_not_found)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/backends/omegaconf/__init__.py", line 84, in render_targets
    raise OmegaConfRenderingError("Error while loading the OmegaConf inventory")
kapitan.inventory.backends.omegaconf.OmegaConfRenderingError: Error while loading the OmegaConf inventory

Note: kapitan command is an function for the below:

kapitan () {
	docker run -t --rm -u $(id -u) -v $(pwd):/src:delegated kapicorp/kapitan:0.34.4-linux-arm64 $@
}

tree of my inventory:

> tree inventory
inventory
├── classes
│   ├── common
│   │   ├── aws.yaml
│   │   ├── azure.yaml
│   │   ├── gcp.yaml
│   │   ├── global.yaml
│   │   └── oci.yaml
│   └── components
│       ├── cert-manager
│       │   └── v1
│       │       └── config.yaml
│       ├── external-dns
│       │   └── v1
│       │       └── config.yaml
│       ├── external-secrets
│       │   └── v1
│       │       └── config.yaml
│       ├── metrics-server
│       │   └── v1
│       │       └── config.yaml
│       └── velero
│           └── v1
│               └── config.yaml
└── targets
    └── clusters
        ├── aws-us-east-1-dca-wl-lab-002.yml
        ├── aws-us-west-2-dca-wl-prd-001.yml
        ├── aws-us-west-2-dca-wl-prd-002.yml
        └── azr-westus-dca-wl-prd-001.yml

16 directories, 14 files

0 replies

ademariag · 2025-01-09T15:56:18Z

ademariag
Jan 9, 2025
Maintainer

I think you have inconsistently named the classes with .yaml,

try changing them to yml instead

see

class file not found for class common.global, tried ['./inventory/classes/common/global/init.yml', './inventory/classes/common/global.yml', './inventory/classes.yml', './inventory/classes/init.yml']
aws-us-west-2-dca-wl-prd-001: could not render due to error Class common.global not found

5 replies

jmmclean Jan 9, 2025

Its a great callout, i made those changes and now get the below

> kapitan compile --inventory-backend omegaconf --migrate
azr-westus-dca-wl-prd-001: could not render due to error Unsupported interpolation type cert_manager
    full_key: kapitan.compile[0].output_path
    object_type=dict
aws-us-east-1-dca-wl-lab-002: could not render due to error Unsupported interpolation type cert_manager
    full_key: kapitan.compile[0].output_path
    object_type=dict
aws-us-west-2-dca-wl-prd-002: could not render due to error Unsupported interpolation type cert_manager
    full_key: kapitan.compile[0].output_path
    object_type=dict
aws-us-west-2-dca-wl-prd-001: could not render due to error Unsupported interpolation type cert_manager
    full_key: kapitan.compile[0].output_path
    object_type=dict
Error while loading the OmegaConf inventory
Traceback (most recent call last):
  File "/opt/venv/bin/kapitan", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kapitan/cli.py", line 639, in main
    args.func(args)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/cli.py", line 76, in trigger_compile
    compile_targets(
  File "/opt/venv/lib/python3.11/site-packages/kapitan/targets.py", line 42, in compile_targets
    inventory = get_inventory(inventory_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kapitan/resources.py", line 345, in get_inventory
    inventory_backend = backend(
                        ^^^^^^^^
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/backends/omegaconf/__init__.py", line 69, in __init__
    super().__init__(*args, **kwargs, target_class=OmegaConfTarget)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/inventory.py", line 51, in __init__
    self.__initialise(ignore_class_not_found=ignore_class_not_found)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/inventory.py", line 92, in __initialise
    self.render_targets(self.targets, ignore_class_not_found=ignore_class_not_found)
  File "/opt/venv/lib/python3.11/site-packages/kapitan/inventory/backends/omegaconf/__init__.py", line 84, in render_targets
    raise OmegaConfRenderingError("Error while loading the OmegaConf inventory")
kapitan.inventory.backends.omegaconf.OmegaConfRenderingError: Error while loading the OmegaConf inventory

I can get around this error by changing the below (view comments: : -> .) - i would have expected the --migrate functionality to do this

parameters:
  cert_manager:
    component_name: cert-manager

    helm:
      chart_version: v1.15.1

  kapitan:
    vars:
      target: test
    compile:
      - output_path: ${cert_manager:component_name}/kustomize # change to ${cert_manager.component_name}/kustomize
        input_type: jinja2
        input_paths:
          - components/${cert_manager:component_name}/v1/kustomize # change to ${cert_manager.component_name}/kustomize

ademariag Jan 9, 2025
Maintainer

yes I will fix it if you open an issue.
is it working however now?

jmmclean Jan 9, 2025

Let me see if I can manually resolve, then I will open an issue

jmmclean Jan 9, 2025

Created an issue here - #1285

Verified that manually fixing the : -> . notation resulted in a successful kapitan compilation

ademariag Jan 9, 2025
Maintainer

Great, join us on the #kapitan channel on the kubernetes slack if you want to join the community.

https://kapitan.dev/latest/support/

or add me on linkedin https://www.linkedin.com/in/alledm/

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kapicorp

Evaluation of reclass and possible inventory alternatives #1007

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments 7 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

kapicorp

Evaluation of reclass and possible inventory alternatives #1007

MatteoVoges May 10, 2023

Current Solution with reclass

Advantages ✔️

Disadvantages ❌

Your opinion

Inventory alternatives

OmegaConf

Advantages ✔️

Disadvantages ❌

Other

References

Replies: 6 comments · 7 replies

ademariag May 17, 2023 Maintainer

MatteoVoges May 17, 2023 Author

MatteoVoges Jun 16, 2023 Author

Installation and setup

Docker

Manual (Dev Mode)

Usage

uberspot Jun 21, 2023 Maintainer

MatteoVoges Jun 29, 2023 Author

ademariag Jun 27, 2023 Maintainer

jmmclean Jan 9, 2025

ademariag Jan 9, 2025 Maintainer

jmmclean Jan 9, 2025

ademariag Jan 9, 2025 Maintainer

jmmclean Jan 9, 2025

jmmclean Jan 9, 2025

ademariag Jan 9, 2025 Maintainer

MatteoVoges
May 10, 2023

Replies: 6 comments 7 replies

ademariag
May 17, 2023
Maintainer

MatteoVoges May 17, 2023
Author

MatteoVoges
Jun 16, 2023
Author

uberspot
Jun 21, 2023
Maintainer

MatteoVoges Jun 29, 2023
Author

ademariag
Jun 27, 2023
Maintainer

jmmclean
Jan 9, 2025

ademariag
Jan 9, 2025
Maintainer

ademariag Jan 9, 2025
Maintainer

ademariag Jan 9, 2025
Maintainer