Add reload and/or restart functionality to Chains

[concaf]

Feature request

in case of a config change or a change in external factors, users wish to be able to tell chains to reload the configuration OR restart the pod to read latest changes

Use case

our users have reported 2 use cases for this:

• MONGO_SERVER_URL has the creds to connect to mongo db which are rotated at a given cadence; when these creds are rotated, they wish to restart chains by hitting an endpoint (say /restart) so the pod restarts and the new creds are picked up
• there are times when updating chains-config configmap or signing-secrets secret does not result in configuration being picked up immediately by the chains controller pod - this is also evident in the getting start tutorial that says Then restart the controller to ensure it picks up the changes: - hitting a /reload (instead of /restart) will help such cases

[lcarva]

I see config watches being setup for both the taskrun and the pipelinerun controllers. Maybe this is not working as expected?

[concaf]

@lcarva not sure - anyway the driving use case for us is the first one (secret rotation) - do you think restart and/or reload can help users nonetheless?

[lcarva]

do you think restart and/or reload can help users nonetheless?

Not sure. If the secret is mounted as a volume on the pod, then it's up to kubernetes to update the contents of the file. (If the volume mount uses subPath, no updates are received.) There could be delays in propagating the updates for projected volumes. This seems to be a known kubernetes issue. It's outside the scope of the controller.

IIUC, Chains does read the file from the filesystem every time it needs it. So any changes seen by the controller should be picked up automatically.

[concaf]

wdyt about the first use case (the more pressing one):

MONGO_SERVER_URL has the creds to connect to mongo db which are rotated at a given cadence; when these creds are rotated, they wish to restart chains by hitting an endpoint (say /restart) so the pod restarts and the new creds are picked up

this has manifested for 2 users - the ask is to be able to restart chains when MONGO_SERVER_URL is rotated externally

MONGO_SERVER_URL env var is usually mounted from a secret, and when the secret rotates the env var is not updated without a pod restart - the ask here is to hit a /restart endpoint on the chains controller when the secret is updated

[lcarva]

I think my previous comment also applies here. If the environment variable is populated from a Secret, then it's outside the scope of the controller to force a refresh of the changes. We would need an approach where the controller is explicitly told which Secret to get the value from. Then it could, possibly, watch the Secret for changes.

Maybe a naive question, but why is killing the Pod not an option? This is probably a better approach as it provides a very specific point in time where a new token is guaranteed to be used.