Problem after migrate from v2.15.0 to v2.17.0 and later

[Josema-Kyndryl]

We have a Zowe v2.15.0 working without problems.

We've replaced it with one v2.17.0 (without modify the yaml configuration file) and now when starting it shows a lot of messages similar to....

ERROR: Error ZWEL0112E: Zowe runtime environment must be prepared first with "zwe internal start prepare" command.
2024-09-03 06:38:45.078 ZWELNCH:16908332 ZWESVUSR INFO ZWEL0004I component gateway(16908331) terminated, status = 28672
2024-09-03 06:38:45.078 ZWELNCH:16908332 ZWESVUSR INFO ZWEL0005I next attempt to restart component gateway in 1 seconds

These are repeated several times until it is finally indicated that the component has not been started....

2024-09-03 06:47:12.633 ZWELNCH:16908332 ZWESVUSR ERROR ZWEL0038E failed to restart component gateway, max retries reached

(The same for app-server, zss, & gateway)

I attach the ZWESLSTC log. ZWESLSTC.SYSLOG.txt

Any ideas to solve it?.

[JoeNemo]

Did the config parms change with correct documentation? Can you include all of your yaml files in this, please?

[Josema-Kyndryl]

Hi Joe.

I attach the configuration file zowe.yaml in ascii
zowe.yaml.txt

[JoeNemo]

"Error ZWEL0112E: Zowe runtime environment must be prepared first with "zwe internal start prepare" command." indicates a missing file called "instance.env" or another, but it's an internal Zowe file. The error message does not indicate this well and should be improved. An earlier step in the launch failed and we are continuing to investigate. @1000TurquoisePogs FYI.

[Josema-Kyndryl]

The error remains after migrate to 2.18.0+20240814

[JoeNemo]

Could you tell us whether "instance.env" or any other assets are missing??

[DivergentEuropeans]

Adding onto @JoeNemo 's comment, we're on the same meeting. So the file may not be called directly "instance.env" but it should be a .env file. Doing a:

find . -name ".env*"

on the workspace folder should find it.

[Josema-Kyndryl]

Hi everyone.
This is the command output:

P083598:/u/P083598:>find /global/zowe -name "*.env*"
/global/zowe/ZOSx/workspace/.env
/global/zowe/ZOSx/workspace/.env/launcher/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/zss/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/zss/.igs.env
/global/zowe/ZOSx/workspace/.env/apiml-common-lib/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/common-java-lib/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/apiml-sample-extension/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/cloud-gateway/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/metrics-service/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/explorer-jes/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/explorer-mvs/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/explorer-uss/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/caching-service/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/files-api/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/gateway/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/app-server/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/app-server/.igs.env
/global/zowe/ZOSx/workspace/.env/jobs-api/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/api-catalog/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/discovery/.instance-igs.env
/global/zowe/ZOSx/workspace/.env/.instance-igs.env
P083598:/u/P083598:>

I supposse the -igs in all files name are caused by the name of HA instances in zowe.yaml (igs$zosa & igs$zosc). I changed it to igs@zosa & igs@zosc and now zowe start with problems.

However, when I try to access the Zowe Desktop I always get the same error in the browser:

[1000TurquoisePogs]

Having a $ in the name probably caused the issue because $ is interpreted by shell scripts for variable substitution. Zowe should either protect itself from use of $ in config or document it as forbidden.

Your current error just means a server didn't start. Check the logs to determine why.

[Josema-Kyndryl]

It's very strange, because the ports are listening:

D TCPIP,,N,CON,SERVER,CLI=ZWE*
EZZ2500I NETSTAT CS V2R5 TCPIP 244
USER ID CONN LOCAL SOCKET FOREIGN SOCKET STATE
ZWESLSTC 002EA56A 0.0.0.0..7556 0.0.0.0..0 LISTEN
ZWE1AC 002EA6FB 0.0.0.0..7552 0.0.0.0..0 LISTEN
ZWE1AD 002EA6D0 0.0.0.0..7553 0.0.0.0..0 LISTEN
ZWE1AG 002EA738 0.0.0.0..7554 0.0.0.0..0 LISTEN
ZWE1CS 002EA615 0.0.0.0..7555 0.0.0.0..0 LISTEN
ZWE1SZ 002EA530 0.0.0.0..7557 0.0.0.0..0 LISTEN
6 OF 6 RECORDS DISPLAYED
END OF THE REPORT

I also see that it is connected against z/OSMF:

D TCPIP,,N,CON,PORT=10443
EZZ2500I NETSTAT CS V2R5 TCPIP 270
USER ID CONN LOCAL SOCKET FOREIGN SOCKET STATE
IZUSVR1 00000038 0.0.0.0..10443 0.0.0.0..0 LISTEN
IZUSVR1 003261B0 192.168.48.41..10443 192.168.48.41..57920 ESTBLSH
ZWE1AG 003261AF 192.168.48.41..57920 192.168.48.41..10443 ESTBLSH
3 OF 3 RECORDS DISPLAYED
END OF THE REPORT

And I don't see any severe errors in the SysLog of the STC:
ZWESLSTC.SYSLOG.txt

[1000TurquoisePogs]

I see

2024-11-12 11:57:29.102 ZWEADS1:https-jsse-nio-0.0.0.0-7553-exec-4:67174539 ZWESVUSR WARN (javax.net.ssl) Ignore impact of unsupported extension: status_request_v2
2024-11-12 11:57:29.108 ZWEAGW1:DiscoveryClient-CacheRefreshExecutor-0:67174684 ZWESVUSR WARN (javax.net.ssl) Unsupported authentication scheme: ecdsa_secp384r1_sha384
2024-11-12 11:57:29.108 ZWEAGW1:DiscoveryClient-CacheRefreshExecutor-0:67174684 ZWESVUSR WARN (javax.net.ssl) Unsupported authentication scheme: ecdsa_secp521r1_sha512
2024-11-12 11:57:29.108 ZWEAGW1:DiscoveryClient-CacheRefreshExecutor-0:67174684 ZWESVUSR WARN (javax.net.ssl) Unsupported authentication scheme: ed448
2024-11-12 11:57:29.108 ZWEAGW1:DiscoveryClient-CacheRefreshExecutor-0:67174684 ZWESVUSR WARN (javax.net.ssl) Unsupported authentication scheme: rsa_pss_rsae_sh

and

2024-11-11 11:55:30 ZWELS:83951763 ZWESVUSR DEBUG (zwe-internal-start-prepare,configure_components) export JAVA_HOME="/usr/lpp/java/J17.0_64"

I'm guessing you didn't just upgrade Zowe, but also switched to java 17.
The doc does not say this is allowed: https://docs.zowe.org/v2.18.x/user-guide/systemrequirements-zos#java

It says java 8 is required.

@balhar-jakub can java 17 be used in zowe v2? I thought it was a goal for more recent revisions of Zowe v2 and I see the servers come up, but the TLS stuff looks confused.

[Josema-Kyndryl]

Hi everyone

I switched to Semeru 17 to see if the problem was solved.

With Java 8 we don't get those SSL errors, but the result is the same:

This is the STC SysLog:
ZWESLSTC.JAVA8.SYSLOG.txt

I don't see any severe errors, only warnings.

[Josema-Kyndryl]

We've migrated to Zowe version 3.0.0+20240919 (and Java-17) but the problem persists, we are not able to access Zowe Desktop.

It shows us correctly the https://zosx.es.kyndryl.com:7554/ page, but when we try with https://zosx.es.kyndryl.com:7554/zlux/ui/v1/ZLUX/plugins/org.zowe.zlux.bootstrap/web/index.html it returns us...

{"messages":[{"messageType":"ERROR","messageNumber":"ZWEAO500E","messageContent":"The service has encountered a situation it doesn't know how to handle. Please contact support for further assistance. More details are available in the log under the provided message instance ID","messageKey":"org.zowe.apiml.common.internalServerError"}]}

And the ZWESLSTC log shows error messages ZWEAT503E, ZWEAT505E and ZWES1606W
ZWESLSTC.SYSLOG.txt

[JoeNemo]

Were there any issues presented when you did the V3 migration, per this doc:

https://docs.zowe.org/stable/whats-new/zowe-v3-migration/

[1000TurquoisePogs]

@Josema-Kyndryl your log of v2 with java 8 from here looks great actually:
#3986 (comment)
I see no errors to explain what you see in the browser, but this is close to good so I'm hoping we can investigate this one further. I would suggest enabling gateway debug (components.gateway.debug) on it to see why its failing since the logs otherwise indicate networking between the servers is succeeding.
I would review the output before sending that as you want to be careful about debug logs - you don't want sensitive headers in the output.

#3986 (comment)
This one seems in worse shape but the logs get cut off. I'm unfamiliar with these messages but sounds like a certificate verification issue may be the cause:

 2024-12-09 09:37:19.551 <ZWEAGW1:parallel-2:33620024> �Ì35mZWESVUSR�Ì0;39m �Ì36mERROR�Ì0;39m ((o.z.a.s.c.v.TrustedCertificatesProvider)) ZWEAT503E An error occurred during retrieval of trusted certificates from the central Gateway. Err
 2024-12-09 09:37:19.552 <ZWEAGW1:parallel-2:33620024> �Ì35mZWESVUSR�Ì0;39m �Ì36mERROR�Ì0;39m ((o.z.a.s.c.v.CertificateValidator)) ZWEAT505E Incoming request certificate is not one of the trusted certificates provided by the central Gat

[Josema-Kyndryl]

I've enabled components.gateway.debug and now the ZWESLSTC log shows Java errors apparently related to DNS: ZWESLSTC.DEBUG.SYSLOG.txt

This z/OS test system does not use DNS, everything is solved by definitions in HOST.LOCAL

Could that be the root of the problem?