z/OSMF static definitions accumulates if configuration is changed

[arxioly]

Describe the bug

New zosmf static definition files are created every time Zowe is started. The old files never removed, so in following scenarios they are accumulating. This can create problems like having multiple ibmzosmf services registered with APIML.

1. Zowe started once as a single insnance (non-HA) with non-HA z/OSMF defined in zowe.yaml. Just to try-it-out. But later the configuration is switched to HA with z/OSMF in HA (different hostname).
2. Zowe started with different haInstances names, then they were defined in zowe.yaml before.
3. Zowe started with different values (could be different from name defined in zowe.yaml) of the HAINST parameter set during start of the instance (/S ZWESVSTC,HAINST=).

In all these cases in api-defs directory, there will be multiple discovery.zosmf_static_definition_yaml_template.<system>.yml with different systems in the name. As a result, all of them will be registered with APIML except those that have same hostname and port.

Some of our customers was facing scenario 1.

The issue is related to #3328 and #3566

Expected behavior
During the startup the existing zosmf static definition files are removed before the creation of new one.

[1000TurquoisePogs]

This is a problem that even predates zwe.
Currently zwe populates the static directory by reading content from a manifest.
So, it should be able to delete content that matches a pattern of things it put there.

That's to avoid deleting sideloaded content. For example, app-server conditionally manually copies a zss static definition into the folder when running in a container because in that case zss isnt around to have its own manifest seen.
Or, if a sysprog today wants to add a server to zowe, they can just copy a file in without worrying about constructing a full zowe extension package with a manifest.

Not sure what our stance on sideloaded content is but I think we can retain ability to do that.

[1000TurquoisePogs]

Looks like today when discovery has file zosmf-static-definition.yaml.template from https://github.com/zowe/api-layer/blob/v3.x.x/discovery-package/src/main/resources/manifest.yaml#L32 it becomes discovery.zosmf_static_definition_yaml_template.<system>.yml because of ${STATIC_DEF_DIR}/${componentName}.${sanitizedDefName}.${zweCliParameterHaInstance}.yml here https://github.com/zowe/zowe-install-packaging/blob/v2.x/staging/bin/libs/component.ts#L420

so how about coding a rm ${componentName}.${sanitizedDefName}.*.yml right before the copy in the above link?

[1000TurquoisePogs]

Actually, what I described is incomplete for HA.
Its possible to have multiple servers intentionally. So, you can't just rm ${componentName}.${sanitizedDefName}.*.yml
Instead you must delete only those that aren't involved in the current HA environment.

For example,

You run 2 HA nodes with ZSS. You do want both ZSS to be registered. You just don't want a 3rd, outdated one from an older config.

You run 2 HA nodes and need z/OSMF. We don't have a way to specify multiple z/OSMF, so there should only ever be one?!

[balhar-jakub]

It's important to distinguish between service and instance. There is 1:n relationships between service and instances. As such for HA setup we want to have one service with multiple instances for z/OSMF and ZSS. It's a bit tedious to do with the static definitions as the original intention was more focused on the dynamic registration.

So this brings question on what types of status definitions should Zowe produce itself and what tooling should be available to rebuild them with new values.