Resize first node of first control plane group

[HendrikLevering]

I have been successfully running a cluster for 1 year. Now my control plane nodes are starting to lack memory. Is it possible to resize the control plane nodes (e.g. from CAX11 to CAX21). I know that I could create a new node pool for all the other nodes. However, what is about the first node from the first control plane node pool? Can I resize that one?

[idebeijer]

I'm not sure about resizing, but I changed my single node control plane from CX32 to CAX21. This is what I did:

1. Add a new control plane nodepool with the desired server_type.
2. After the new control plane node joined, SSH to the new control plane node and create an on-demand etcd snapshot on it. k3s etcd-snapshot save, just in case so I can always do a restore.
3. SSH into the new and old control plane nodes and stop the k3s service on both of them: systemctl stop k3s
4. Start k3s on the new control plane with a cluster reset: k3s server --cluster-reset. This means that the new control plane will forget about the other old one, and thus allowing it to start by itself as the only member. You'll likely have to remove the key and its value "server" from config.yaml (by default in /etc/rancher/k3s/
5. After starting k3s with a cluster reset, as it will print out, you'll have to start it again without the --cluster-reset flag. So either just k3s server or systemctl start k3s should do it.

Keep in mind this was with just 1 control plane node as target, more steps are probably needed for a control plane with multiple nodes.

The most nodes at one time I had was 2, as I just moved from a single node, temporarily to a 2 node control plane and then back to single node on the new instance type. I imagine the process should be similar for a multi-node control plane setup (if you can tolerate control plane downtime), but you'll likely need to do some extra steps since you have more nodes, because you'll still only run k3s server --cluster-reset on one of the new nodes. As for the other new nodes, you'll likely have to delete the data directory before starting them again. More on that in the k3s docs: https://docs.k3s.io/cli/etcd-snapshot?etcdsnap=High+Availability.

[HendrikLevering]

thanks for your response. That sounds like a wild adventure for a production cluster.

[idebeijer]

Maybe you could try on a non-production cluster and just add another node pool for the control plane with e.g. 3 nodes if you were coming from 3 nodes. And then after they have joined successfully, delete the old 3 nodes and see if that works out fine.

If etcd can handle that, then you'll have migrated the control plane to the new instance types.