microk8s reset does not completely reset the cluster

[LoniasGR]

Summary

When doing microk8s reset, validatingwebhookconfigurations.admissionregistration.k8s.io are not deleted. This can lead to weird errors, as mentioned also here. In my case, the problem was with the ingress-nginx-controller.

What Should Happen Instead?

microk8s reset should delete all previously resources, effectively providing a fresh cluster.

Reproduction Steps

1. Setup microk8s cluster.
2. Install ingress-nginx-controller as described here.
3. Run microk8s reset
4. Run microk8s.kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io

There should be still an ingress-nginx-admission webhook.

Introspection Report

I don't think it's needed.

Can you suggest a fix?

The problem is that here, from the non-namespaced API resources, only crds, priority classes, storage classes and namespaces are deleted. I'm not exactly sure which of the non-namepsaced API should be cleaned up, but probably there are more than just validating webhooks.

Are you interested in contributing with a fix?

Probably, if I was informed what resources should be actually deleted.