Support for custom auto-deploying manifests

[captnCC]

As described already by @mysticaltech here k3s supports automatically adding custom manifests by deploying them to /var/lib/rancher/k3s/server/manifests on the init node. Docs

From here on I see 2 paths for the implementation:

• Deploy all files found in a extra-manifests directory straight to the server
• Provide a variable on the module to set a list of files to deploy them to the server

The first one would be pretty simple and easy to use/implement. But in my opinion it lacks support for customisation of the manifests in terraform (to set some variables etc). Or correct me if it is possible with terraform to run them through a template engine before running the cluster module itself.

The second option would be more complicated, but leaves much more room for customisation of the files before pushing them.
This would open the possibility of using the complete yaml toolset that terraform offers before encoding and placing it on the server.

Or just adding support for both 🤷🏻‍♂️

While looking at the existing code for deploying the manifest for stuff like longhorn, cert-manager, hetzner-csi, I wonder if there is a reason they are placed into /var/post-install and executed manual instead of the taking the route of the auto-deploying manifest?

[mysticaltech]

@captnCC You are right; we could have generated them and placed them inside the auto-deploy folder. But at the time, we did not realize that was possible, and it also lacks the advantage of controlling the sequence. But definitely, some things could go there!

To answer your first question, YES, generating those via Terraform would work and allow for more customizations. But it's really in our objective to not support third parties utilities that are not directly related to "deploying an optimized cluster" which we try to do first, and would add more maintenance load.

So, for now, the extra-manifest route would be what I would choose. But am always open to hearing more arguments if you think the second method is really needed (from my current vantage point, I doubt we need many values from the cluster for most packages).

[phaer]

Thanks for taking this on, @captnCC!

The first one would be pretty simple and easy to use/implement. But in my opinion it lacks support for customisation of the manifests in terraform (to set some variables etc). Or correct me if it is possible with terraform to run them through a template engine before running the cluster module itself.

I do believe this should be possible by using fileset together with templatefile and will gladly try to help if necessary (either here, or ping me at [email protected] if you use matrix).

[mysticaltech]

SORRY IN ADVANCE FOR THE EVENTUAL TYPOS, WRITING FROM MOBILE AND IT'S LATE.

@phaer You are right fileset seems ideal along with templatefile. BUT we need to make this as generic as possible! So snatching the whole folder, as k3s does it seem ideal.

Now, if we could also support an extra-manifest-kustomization folder, and generate the extra-manifest folder from it, to be copied. That would be ideal IMHO.

Now for added sugar, if that kustomization file can be a template with access to all definable set of variables from terraform, defined as hash for instance,

extra_manifest_kustomizatiom_params = {"name_in_kustomization": "name_of_terraform_ressource", ... }

I guess that would make it generic enough! So basically in this case, we would add the generates (from kustomization) file to the auto-deploy folder. Only one file!

The beauty of that latter approach is that it supports all kinds of yaml files including urls,, and local HelmChart definition, it really doesn't matter.

So maybe best to just force the user to use that latter approach.

To recap, kustomization from extra-manifests, with optional use of template variables, via their definition via a terraform var (object that does the mapping). The kustomize file in the extra-manifest folder would need to be named something like kustomize.yaml .tpl. And have all local needed yaml files next to it in the same folder.

Even simpler, would be to just kubectl -k apply that folder into the cluster, after copying the whole folder and with the final kustomize.yaml to the node. FORGETTING ABOUT THE AUTO-DEPLOY folder, as it's out of our control and harder to debug!!

Ideally, THIS SHOULD BE THE VERY LAST THING THAT The cluster does as part null ressource like @phaer proposed. So that if errors are present it does not affect the core cluster logic. And would allow for iteration by the user to make it work!

@captnCC if you decide to take this idea on, you could inspire yourself from the current null_ressource.kustomization that we use.

[captnCC]

I've never used kustomize before but it looks pretty capable for what we want to do.
So just to quickly summarise the steps:

• Look for the kustomize.yaml or kustomize.yaml.tpl file
• [If its templated, push it though the templatefile function with extra_manifest_kustomizatiom_params]
• Copy it together with the remaining files from the extra-manifests directory folder to the init control-plane
• Apply them all with kubectl -k apply

Without experience and out of personal taste, I would even prefer to add the kustomize.yaml not as file or template but as pure object like it is constructed in init.tf. So this would add total freedom to the user, how to provide the values (as variable, yamldecode, templatefile etc ) or don't provide one at all (in which case we could place a default/empty one on the server).

[mysticaltech]

That's about right @captnCC. Kustomization is super powerful! And don't forget to run it in a null_ressource that executes at the very end (through setting the right dependency constraints).

I understand your preference to use yamlencode; but do not know how that code work in a generic way. If you know how, please do propose a flow for this.

As for supporting kustomize.yaml, I guess kustomize.yaml.tpl is enough, as if extra_manifest_kustomizatiom_params is empty, it will just generate an equal kustomize.yaml from it!

[captnCC]

@mysticaltech Terraform already cancelled my approach, since it doesn't support untyped objects as variables. Hence it is impossible to pass the values of a kustomize.yaml to the module.

Right now I've got a basic implementation of the template flow running! Just a bit more polishing and I will provide a PR for it

Thanks @mysticaltech and @phaer for your input on it!

[mysticaltech]

Awesome!!

[otavio]

Consider using fluxcd so it could be used to allow for cluster upgrade once deployed.

[mysticaltech]

Thanks @otavio , I understand where you are coming from, but it's out of scope for that feature. Maybe for a future one on automatically updating default charts. But seems risky! I prefer we just let everyone choose how to do that. However an addition to our examples folder on how to use flux to do the custom updates of charts would be nice. (The cluster itself along with Traefik are already automatically upgraded via the system upgrade controller).

[mysticaltech]

The feature was released in v1.5.6, thanks to @captnCC! 🎯