ManageIQ Quinteros-1 - VMware vSphere deployment not working #23068

erwinb3 · 2024-06-14T13:15:01Z

erwinb3
Jun 14, 2024

First of all thanks for the work done to achieve the new update Quinteros-1.

Unfortunately we cannot get the new version Quinteros-1 to run on the VMware environment, the old version (manageiq-vsphere-petrosian) works without any problems.

Simple setup:
-1 Configure Network settings
-5 Configure application > 1. Create key > 1. Create Internal Database> 3. make no messaging changes> 1. Database disk /dev/sdb: 153600mb> Should run as standalone database? No
Region 44

After this the service should start but it gets stuck on starting manageiq

services:
kafka failed to start
"top(1) data collection" failed to start

https://www.manageiq.org/docs/reference/latest/installing_on_vmware_vsphere/

kodkurdu · 2024-06-16T16:42:37Z

kodkurdu
Jun 16, 2024

Hello team,

I'm having the same problems and looking for the installation guide. I searched the logs and found the following. I was expecting Manageiq to manage this certificate issue.

Jun 16 19:32:32 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:32,879] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)
Jun 16 19:32:32 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:32,987] INFO [Controller id=0, targetBrokerId=0] Failed authentication with manageiq.test.info/192.168.90.4 (channelId=0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
Jun 16 19:32:32 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:32,987] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.90.4 (channelId=192.168.90.4:9093-192.168.90.4:37162-249) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
Jun 16 19:32:32 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:32,987] INFO [Controller id=0, targetBrokerId=0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient)
Jun 16 19:32:32 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:32,987] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (manageiq.test.info/192.168.90.4:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
Jun 16 19:32:32 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:32,988] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)
Jun 16 19:32:33 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:33,097] INFO [Controller id=0, targetBrokerId=0] Failed authentication with manageiq.test.info/192.168.90.4 (channelId=0) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
Jun 16 19:32:33 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:33,098] INFO [Controller id=0, targetBrokerId=0] Node 0 disconnected. (org.apache.kafka.clients.NetworkClient)
Jun 16 19:32:33 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:33,098] INFO [SocketServer listenerType=ZK_BROKER, nodeId=0] Failed authentication with /192.168.90.4 (channelId=192.168.90.4:9093-192.168.90.4:37170-249) (SSL handshake failed) (org.apache.kafka.common.network.Selector)
Jun 16 19:32:33 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:33,098] ERROR [Controller id=0, targetBrokerId=0] Connection to node 0 (manageiq.test.info/192.168.90.4:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
Jun 16 19:32:33 manageiq.test.info kafka-server-start.sh[1452]: [2024-06-16 19:32:33,099] INFO [Controller id=0, targetBrokerId=0] Client requested connection close from node 0 (org.apache.kafka.clients.NetworkClient)

0 replies

bdunne · 2024-06-17T15:06:03Z

bdunne
Jun 17, 2024
Maintainer

The errors in the logs indicate that the hostname entered during messaging configuration does not match the hostname of the appliance. Have you set the hostname and added it to /etc/hosts on the appliance?

0 replies

erwinb3 · 2024-06-17T15:35:34Z

erwinb3
Jun 17, 2024
Author

Appliance Hostname

[root@manageiq-test log]# cat /etc/hosts
127.0.0.1        manageiq-test.mydomain.nl localhost localhost.localdomain localhost4 localhost4.localdomain4
::1              manageiq-test.mydomain.nl localhost localhost.localdomain localhost6 localhost6.localdomain6

Welcome to the ManageIQ Virtual Appliance.

To modify the configuration, use a web browser to access the management page.
Hostname:                manageiq-test.mydomain.nl


Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: Caused by: java.io.IOException: Configuration Error:
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011Line 4: expected [option value], found [null]
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.match(ConfigFile.java:579)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:480)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:427)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011at sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
Jun 17 11:32:37 manageiq-test kafka-server-start.sh[4882]: #011... 16 more
Jun 17 11:32:37 manageiq-test systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE
Jun 17 11:32:37 manageiq-test systemd[1]: kafka.service: Failed with result 'exit-code'.
Jun 17 11:32:37 manageiq-test systemd[1]: kafka.service: Service RestartSec=100ms expired, scheduling restart.
Jun 17 11:32:37 manageiq-test systemd[1]: kafka.service: Scheduled restart job, restart counter is at 5.
Jun 17 11:32:37 manageiq-test systemd[1]: Stopped kafka service.
Jun 17 11:32:37 manageiq-test systemd[1]: Started kafka service.
Jun 17 11:32:38 manageiq-test kafka-server-start.sh[5309]: [2024-06-17 11:32:38,683] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
Jun 17 11:32:38 manageiq-test manageiq-messaging-ready[3127]: manageiq-test.mydomain.nl 9093 - not accepting connections
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: [2024-06-17 11:32:39,004] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: [2024-06-17 11:32:39,099] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: org.apache.kafka.common.KafkaException: Exception while loading Zookeeper JAAS login context [java.security.auth.login.config=/opt/kafka/config/kafka_server_jaas.conf, zookeeper.sasl.client=default:true, zookeeper.sasl.clientconfig=default:Client]
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:67)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at kafka.server.KafkaServer$.zkClientConfigFromKafkaConfig(KafkaServer.scala:80)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at kafka.server.KafkaServer.<init>(KafkaServer.scala:150)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at kafka.Kafka$.buildServer(Kafka.scala:73)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at kafka.Kafka$.main(Kafka.scala:87)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at kafka.Kafka.main(Kafka.scala)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: Caused by: java.lang.SecurityException: java.io.IOException: Configuration Error:
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011Line 4: expected [option value], found [null]
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:137)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile.<init>(ConfigFile.java:102)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at java.lang.Class.newInstance(Class.java:442)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at javax.security.auth.login.Configuration$2.run(Configuration.java:255)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at javax.security.auth.login.Configuration$2.run(Configuration.java:247)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at java.security.AccessController.doPrivileged(Native Method)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at javax.security.auth.login.Configuration.getConfiguration(Configuration.java:246)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at org.apache.kafka.common.security.JaasUtils.isZkSaslEnabled(JaasUtils.java:63)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011... 5 more
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: Caused by: java.io.IOException: Configuration Error:
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011Line 4: expected [option value], found [null]
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.ioException(ConfigFile.java:666)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.match(ConfigFile.java:579)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.parseLoginEntry(ConfigFile.java:480)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.readConfig(ConfigFile.java:427)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:329)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.init(ConfigFile.java:271)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011at sun.security.provider.ConfigFile$Spi.<init>(ConfigFile.java:135)
Jun 17 11:32:39 manageiq-test kafka-server-start.sh[5309]: #011... 16 more
Jun 17 11:32:39 manageiq-test systemd[1]: kafka.service: Main process exited, code=exited, status=1/FAILURE
Jun 17 11:32:39 manageiq-test systemd[1]: kafka.service: Failed with result 'exit-code'.
Jun 17 11:32:39 manageiq-test systemd[1]: kafka.service: Service RestartSec=100ms expired, scheduling restart.
Jun 17 11:32:39 manageiq-test systemd[1]: kafka.service: Scheduled restart job, restart counter is at 6.
Jun 17 11:32:39 manageiq-test systemd[1]: Stopped kafka service.
Jun 17 11:32:39 manageiq-test systemd[1]: kafka.service: Start request repeated too quickly.
Jun 17 11:32:39 manageiq-test systemd[1]: kafka.service: Failed with result 'exit-code'.
Jun 17 11:32:39 manageiq-test systemd[1]: Failed to start kafka service.
Jun 17 11:32:48 manageiq-test manageiq-messaging-ready[3127]: manageiq-test.mydomain.nl 9093 - not accepting connections
Jun 17 11:32:58 manageiq-test manageiq-messaging-ready[3127]: manageiq-test.mydomain.nl 9093 - not accepting connections
Jun 17 11:33:08 manageiq-test manageiq-messaging-ready[3127]: manageiq-test.mydomain.nl 9093 - not accepting connections
Jun 17 11:33:19 manageiq-test manageiq-messaging-ready[3127]: manageiq-test.mydomain.nl 9093 - not accepting connections
Jun 17 11:33:29 manageiq-test manageiq-messaging-ready[3127]: manageiq-test.mydomain.nl 9093 - not accepting connections

0 replies

nasark · 2024-06-18T14:33:38Z

nasark
Jun 18, 2024

Hi @erwinb3 and @kodkurdu, messaging must be configured with a FQDN hostname that is not localhost/127.0.0.1. From the docs:

It is recommended to use your Fully Qualified Domain Name (FQDN) as the messaging hostname rather than localhost, ensure that a resolvable and reachable non-localhost name entry is present in /etc/hosts

For example:

[root@nasark ~]# cat /etc/hosts
<my-ip> nasark.test.raleigh.com
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@nasark ~]# hostname -f
nasark.test.raleigh.com

and I should be able to ping the ip/hostname

[root@nasark ~]# ping nasark.test.raleigh.com
PING nasark.test.raleigh.com (<my-ip>) 56(84) bytes of data.
64 bytes from nasark.test.raleigh.com (<my-ip>): icmp_seq=1 ttl=64 time=0.620 ms

3 replies

nasark Jun 18, 2024

Let me know how it goes for you

erwinb3 Jun 18, 2024
Author

Hi @nasark
Hostname and network settings are correct.
Dns / ping etc is working and we are using the fqdn name.
The same (basic) settings works fine on the old appliance petrosian.
I suspect that there is a problem in this new vSphere image.

agrare Jun 18, 2024
Maintainer

@erwinb3 do you have your hostname resolving to a non-127-class IP address?
You /etc/hosts file above shows 127.0.0.1 manageiq-test.mydomain.nl and kafka/zookeeper SSL doesn't work with a loopback address unfortunately.

Can you run nslookup $(hostname) and verify what the hostname is resolving to?

erwinb3 · 2024-06-18T20:11:59Z

erwinb3
Jun 18, 2024
Author

Manual edited the hosts file and add manageiq-test

This is working on manageiq-vsphere-petrosian...

tried to reconfigure the message server without success

sticks around: starting ManageIQ server......

Logging

1 reply

erwinb3 Jun 20, 2024
Author

@nasark and @agrare @bdunne
I would like to switch to the new version.
Do you have any idea what the problem could be?
Thnx

kodkurdu · 2024-06-24T07:58:33Z

kodkurdu
Jun 24, 2024

I applied same config on manageiq-morphy and it runs.

7 replies

nasark Jun 25, 2024

I was not able to reproduce the issue on Quinteros-1. I followed the steps I outlined above with regards to FQDN and non-localhost

erwinb3 Jun 25, 2024
Author

Thnx for testing...
It turns out that if I do not change the password after the rollout of the appliance and set (passwd root to smartvm) and use smartvm as the passwordeverywhere, it works.

If I deviate from this password, an error occurs.

Working now...:)

Fryguy Jun 26, 2024
Maintainer

It turns out that if I do not change the password after the rollout of the appliance and set (passwd root to smartvm) and use smartvm as the passwordeverywhere, it works.

That's an interesting find.

jrafanie Jul 24, 2024
Maintainer

@nasark Can you take a look? @agrare said it might be related to specific characters we had issues with in ManageIQ/manageiq-appliance_console#255. Thoughts?

nasark Jul 24, 2024

@erwinb3 Did the original keystore password you were using (before smartvm) contain any special characters? There is a JKS (Java Keystore) limitation that prevents most special characters. We now enforce that with ManageIQ/manageiq-appliance_console#255

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ManageIQ

ManageIQ Quinteros-1 - VMware vSphere deployment not working #23068

{{title}}

Replies: 6 comments 11 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

ManageIQ Quinteros-1 - VMware vSphere deployment not working #23068

Replies: 6 comments · 11 replies

bdunne Jun 17, 2024 Maintainer

erwinb3 Jun 17, 2024 Author

erwinb3 Jun 18, 2024 Author

agrare Jun 18, 2024 Maintainer

erwinb3 Jun 18, 2024 Author

tried to reconfigure the message server without success

Logging

erwinb3 Jun 20, 2024 Author

erwinb3 Jun 25, 2024 Author

Fryguy Jun 26, 2024 Maintainer

jrafanie Jul 24, 2024 Maintainer

Replies: 6 comments 11 replies

bdunne
Jun 17, 2024
Maintainer

erwinb3
Jun 17, 2024
Author

erwinb3 Jun 18, 2024
Author

agrare Jun 18, 2024
Maintainer

erwinb3
Jun 18, 2024
Author

erwinb3 Jun 20, 2024
Author

erwinb3 Jun 25, 2024
Author

Fryguy Jun 26, 2024
Maintainer

jrafanie Jul 24, 2024
Maintainer