Customize the deployment of cert-manager installed via OLM

[hongkailiu]

Follow up a slack consulting.
https://kubernetes.slack.com/archives/C4NV3DWUC/p1619141623473100

If I understand the cert-manager operator correctly, we still need to modify deployment manifests for customizations, such as args of the container such as --dns01-recursive-nameservers="8.8.8.8:53", env vars of the container providing aws credentials, and replicas of the webhook and affinity.

It would be great if the operator can read those customizations from a configMap and reconcile the related manifests.

[kfox1111]

I think subscriptions can inject env vars into the resulting deployment. maybe it can be configured that way, or maybe that functionality can be added?

[MajorFlamingo]

The customization is also needed to inject custom RootCAs if using a local ACME server, tracked here: cert-manager/cert-manager#2332

It looks like a drawback that we cannot configure additional args/mounts.

[wallrj]

@hongkailiu and @MajorFlamingo please read Configuring Operators deployed by OLM and see if any of those can be used to achieve your use-cases.

It might be that we need to update the cert-manager Deployments to allow command line arguments of the cert-manager components to be supplied from environment variables. If so, let me know, and I can either modify the Deployment here in the OLM package or upstream in cert-manager.

Please also comment on the RedHat OCP UI proposal Subscription Config when installing an Operator.

[sipe90]

It would be great if I could pass the --dns01-recursive-nameservers-only argument to the operator but it doesn't seem to be possible. Also there is no way to set the log level since AFAIK the only way to set it is with the --v argument.

This is the only thing preventing me from using the operator at the moment. As an alternative I've resorted to using the Helm chart instead but the operator would be way more convenient to use in OpenShift.

[bdurrow]

@wallrj, I poked at this and I don't see a way to do it presently. The most straight forward would be to allow for an environment variable that would be used as arguments to the controller. Replacing the existing controller entrypoint with one that called the controller with any additional arguments from that environment variable. Alternatively the controller could be modified to accept some configuration from environment variables but this seems like it would need to touch the code in several different places unless there is a way to do it for using for all options (perhaps using this method).

Requiring that all Issuers use the same values for thing like Recursive DNS server seems like it might be limiting and perhaps changing the CRDs to allow for specifying more of these options in the CR would be a nice target for the future but it seems like the path to that solution would be considerably longer than these other two approaches.

[raelga]

Any update on this matter? :)

[wallrj]

@bdurrow , @raelga and anyone else interested in this, please review

• Document how to configure cert-manager via OLM website#983
  ...my attempt to document the current state of things.

[raelga]

Hey @bdurrow,

Thanks for the updated documentation.

For our use case, we ended developing a new operator using the OperatorSDK Helm framework. It's actually a kind of meta-operator, when the operator image is built, it fetches the helm chart and stores it into the container image.

When deployed into a cluster, it allows every single configuration available in the helm chart using an OperatorConfig instance manifest to define the values.yaml. With that configuration, it renders, deploys and reconciles all the resources from the helm chart.

The code is available here:

https://github.com/3scale-ops/cert-manager-helm-operator

Images are built on GH actions:

https://github.com/3scale-ops/cert-manager-helm-operator/actions

And the images are pushed to quay.io:

• quay.io/3scale/cert-manager-helm-operator-catalog
• quay.io/3scale/cert-manager-helm-operator-bundle
• quay.io/3scale/cert-manager-helm-operator

A few things about the implementation:

• When a new chart is available, we only need to run the download-helm-chart makefile target with the new VERSION and build the container operator (container-build), catalog and bundles images (bundle-publish). The helm chart code doesn't even needs to be in the repo, is fetched on build time in the GH action:

https://github.com/3scale-ops/cert-manager-helm-operator/runs/6691411489?check_suite_focus=true

https://github.com/3scale-ops/cert-manager-helm-operator/blob/main/Makefile#L261-L276

An example on how a release looks like is this PR https://github.com/3scale-ops/cert-manager-helm-operator/pull/3/files. Everything in the bundle path is generated from the yamls in config. And most part of those config yamls are also generated from the information available on the helm charts. We added some changes like missing rbac permissions, unauthenticated metrics and enforcing inject-apiserver-ca: true.

• CRDs are managed by OLM and shipped with the bundle, so InstallCRDs option is set to false by default.

• All the configurations available on the values.yaml are now available on the OperatorConfig resource:

apiVersion: operatorconfigs.operator.cert-manager.io/v1alpha1
kind: OperatorConfig
metadata:
  name: cluster
spec:
  prometheus:
    enabled: true
    servicemonitor:
      enabled: true
      targetPort: 9402
      path: /metrics
      interval: 30s
  podDnsConfig:
    nameservers:
      - 1.1.1.1
      - 8.8.8.8
  resources:
   requests:
     cpu: 10m
     memory: 96Mi
   limits:
     cpu: 100m
     memory: 256Mi

[QuingKhaos]

@wallrj this operator needs a extra CertManager CRD which enables to install cert-manager driven via the CRD. IMHO the operator should never install the software it manages directly as a result from the subscription, but provide a CRD to install an instance of the software.

A CertManager CRD can expose all configuration options cert-manager and the components provided, but is still opinionated in the exposed configurations. Example would be the ArgoCD operator. The OLM Subscription config is solely to configure the operator-controller, not the operands an operator deploys.

In the end, this miss is a reason, why the new cert-manager Operator for OpenShift is a separate implementation (providing a CertManager CRD) and not a fork of the upstream operator, like most Red Hat provided operators are.

[hongkailiu]

Thanks for all the input and effort on the topic.

Some update from my side:

• Removing `--dns01-recursive-nameservers="8.8.8.8:53": cert-manager b02: remove flag/dns01-recursive-nameservers openshift/release#34942 and cert-manager: remove all flag/dns01-recursive-nameservers openshift/release#34944
• Removing env. var. for AWS credentials: cert-manager: set up AWS credentials on ClusterIssuers openshift/release#34947

The only customization is the webhook (replicas and affinity) which is not a blocker (still nice to have though) for us.

We will start to adopt cert-manager-operator after it is GA (currently still in TechPreview).

[absynth76]

Just a fyi, but redhat cert-manager operator (openshift) adopted the CertManager CR solution described above #22 (comment)