BPXP005I A FORK OR SPAWN ERROR WAS ENCOUNTERED. RETURN CODE 00000070

[bobbydixon]

Describe the bug

Following an IPL introducing a round of z/OS 2.4 maintenance (including Zowe PTFs UO90057 and UO90058 (2.16.0)), we received the following fork/spawn error during the Zowe server startup:

BPXP005I A FORK OR SPAWN ERROR WAS ENCOUNTERED. RETURN CODE 00000070 REASON CODE 0B150020
BPXP005I A FORK OR SPAWN ERROR WAS ENCOUNTERED. RETURN CODE 00000070 REASON CODE 0BFC0000

To Reproduce
Steps to reproduce the behavior:

1. Running on z/OS V2.4
2. Installed RSU that included Zowe V2.16 PTFs
3. Start Zowe STCs, and receive the error in ZWESTSTC

Expected behavior
We usually see a bunch of BPXAS address spaces get spawned

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context

We found that after correcting the "pluginLocation:" path in workspace file 'org.zowe.zlux.auth.trivial.json ', We re-IPLed the system and was unable to recreate the original issue. So, it seems the OMVS fork/spawn issue was caused by the missing path??

We only have the following Zowe entries in the SCHEDxx PARMLIB member:

PPT PGMNAME(ZWESIS01) KEY(4) NOSWAP
PPT PGMNAME(ZWESAUX) KEY(4) NOSWAP

We see the following message in the joblog:

ZWES1006E Error while parsing plugin definition file '/RADZ5/usr/lpp/zowe/compon

This appears to be similar to an older github issue #3137 but Andrew suggested creating a new issue, because old issues for old zowe versions may be cleaned up.

[JoeNemo]

I hope that the #3137 issue is checked for explaining this case (or not).

[Andrew-J-Metzger]

Hey team, just to add some context in addition to Bobby's report:

Perhaps Zowe should issue some warning here, if we detect an incorrect key? But, in this case, we believe BPXP005I should not have been issued.

If you are not using KEY(4) for the cross-memory server/ and KEY(8) for the launcher, z/OS should error out, right?

Notice right after the BPXP005I, zss starts anyways. :)

Thanks and make it a great day,
Andrew

[bobbydixon]

I just noticed I mentioned the wrong STC name in my opening statements, it's the ZWESLSTC where we see the two BPXP005I messages just over 50 seconds after the STC is started, sorry about.

@JoeNemo thanks for the feedback. Yes, I checked the referenced github issue, and confirmed the SCHEDxx member is correct, and the client does not want to changed the useConfigmgr: true to 'false' in their zowe.yaml file.

The client talked with their resident Zowe admin who has run into this before and just manually edits the workspace file org.zowe.zlux.auth.trivial.json and changes the "pluginLocation:" field. So, they went ahead and removed the '/RADZ5' from the path in this file, restarted the Zowe server and no longer see those recurring errors.

It would help to understand how the workspace is caching this location and have it record the symbolic link versus the resolved full path name. That would eliminate the need for us to remember to make this change.

[bobbydixon]

Going through the ZWESLSTC joblog, in addition to the 2 x BPXP005I messages, we also see a flood (119 occurrences) of the following messages (I've just cut and pasted the first page:

2024-05-30 18:24:17.700 ZWED:65907 ZWESVUSR WARN (_zsf.bootstrap,plugin-loader
location: '/RADZ5/usr/lpp/zowe/components/app-server/share/zlux-server-framewo
identifier: 'org.zowe.zlux.auth.trivial',
error: ÝError: ENOENT: no such file or directory, open '/RADZ5/usr/lpp/zowe/co
errno: -129,
code: 'ENOENT',
syscall: 'open',
path: '/RADZ5/usr/lpp/zowe/components/app-server/share/zlux-server-framework
}
}

Which the client resolves using the previously mentioned steps.

Any idea why this value is being stored?

[JoeNemo]

@1000TurquoisePogs @struga0258 @DivergentEuropeans any ideas?

[DivergentEuropeans]

@bobbydixon It looks like some of that message is clipped, but the general idea seems to be you have a plugin identifier JSON somewhere that points to a 'org.zowe.zlux.auth.trivial' plugin. This identifier also has a location for this plugin, and this plugin's location doesn't exist. So the solution is to remove the plugin identifier JSON found in:
<zowe.workspaceDirectory>/app-server/plugins

[bobbydixon]

@DivergentEuropeans I was discussing this with my colleague Joe Pun, he suggested renaming the ../workspace directory, then restarting Zowe, which the client did, and just like Joe suspected, the ../workspace directory was recreated, but interestingly enough the file called org.zowe.zlux.auth.trivial.json was not recreated. So we're not sure who/what created the file originally. We're still trying to establish with the client whether the RSE plugin still works after this exercise...

[bobbydixon]

Sean Grady said "trivial auth" (org.zowe.zlux.auth.trivial.json) on the Zowe servers was included in old versions of zowe and never used. It was really there for development and troubleshooting purposes. It's since been removed for new installs, but, if you upgrade from an older install it will persist. The solution is deleting the whole "workspace/app-server/plugins" that causes the folder to be regenerate at next startup, this also helps to clean up outdated stuff.

Case closed....thanks y'all