Volcap not detecting Sensors (remote_eye service) - remote_eye_app works

[cebdionysos]

Hi

First it used to work and stopped working. Since then went to all uninstalling and reinstalling. Still only app will work, but not the remote_eye_service

Setup:
Workstation with Volcap and RabbitMQ (latest version)
4 Kinect on 4 Asus minipc PB60G
Tried, with and without Firewall on, but still all necessary ports have been configured according to docs, both on volcap and sensors
Tried on a single network (Work Domain network)
Then we added a 2.5 G unmanaged switched to create a second private network 192.168.1.x
Tried also with only the 2.5 G network. All 3 combination did not work.

I need help to figure out, why volcap can talk to the remote_eye_services and connect the sensors.

Thanx

Sebastien D.

[ankarako]

Hi,

There are issues when there are > 1 ethernet adapters in the system that hosts volcap. Could you check if volcap sends the first (handshake) message via the correct IP? (It should be noted when volcap starts in the command line)

[cebdionysos]

I will check today. But knowing this could have been a problem I did disabled the domain network so only was (private was left) and still it did not work. Again I will check later today, but how doe it work? Is broadcasting to find the sensors? Can we specify IP in a file instead of broadcasting?

[ankarako]

You could start the remote eyes manually via the command line and define the ip to connect to.

[cebdionysos]

Yes I did that to see that it worked. But the sensors have no monitors. The service alwways run and this would be helpful to connect without intervention. As per app, it means that we start volcap first and need to connect remotely with RDP to start the app, not good with 4 or 5 headless sensors

Let me again check your first question later today and then we could see what next step we could check to find why it is no connecting
regards,

Seb

[ankarako]

Yeah, I know what you mean, we usually used Windows Remote Desktop to operate the headless mini-pcs

[cebdionysos]

When I start Volcap it uses the private Network:

Connecting to broker: { amqp://volumetric:[email protected]:5672}
Exception: send_to: An attempt was made to access a socket in a way forbidden by its access permissions

GL_VENDOR : NVIDIA Corporation
GL_RENDERER : NVIDIA GeForce RTX 3090/PCIe/SSE2
GL_VERSION : 3.2.0 NVIDIA 516.59
GLSL_VERSION : 1.50 NVIDIA via Cg compiler

DDRenderInterfaceCoreGL initializing ...

DDRenderInterfaceCoreGL::setupShaderPrograms()
DDRenderInterfaceCoreGL::setupVertexBuffers()
DDRenderInterfaceCoreGL ready!

And I see it in RabbitMQ connection (server installed on same workstation as Volcap:

192.168.1.1:62132? volumetric running ○ AMQP 0-9-1 1 0 B/s 0 B/s
192.168.1.1:62133? volumetric running ○ AMQP 0-9-1 1 0 B/s 0 B/s
192.168.1.1:62134? volumetric running ○ AMQP 0-9-1 2 0 B/s 0 B/s
192.168.1.1:62135? volumetric running ○ AMQP 0-9-1 1 0 B/s 0 B/s

But I do I know if the services from sensors are using the correct IP since Even when I restart the service it never shows a connection in Rabbit MQ?

[ankarako]

The service writes a log file inside the remote eye's folder. The remote eyes write a log file also. Please, check if the broker's IP is the same with the ones that try to talk to the service.

[cebdionysos]

After restarting service :
[2022-09-23 3:04:17 PM]: Exception: System.Net.Sockets.SocketException (0x80004005): A blocking operation was interrupted by a call to WSACancelBlockingCall
at System.Net.Sockets.Socket.ReceiveFrom(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint& remoteEP)
at System.Net.Sockets.UdpClient.Receive(IPEndPoint& remoteEP)
at remote_eye_service.UDPListener.Listen()
[2022-09-23 3:04:22 PM]: Listener started listening...
[2022-09-23 3:04:22 PM]: Process name: C:\Capturer\bin\remote_eye_k4a

[ankarako]

Hmmm, please follow the following steps:

• Close volcap (if open)
• Restart the service
• Open volcap

(If volcap is open, and then you restart the service, when you close volcap it will say to the service to kill the remote_eye, but the remote_eye is not open, and the service crashes)

[cebdionysos]

BTW there is no sensor (kinect) on the workstation with volcap and RabbitMQ server.

[2022-09-23 3:27:29 PM]: Listener started listening...
[2022-09-23 3:27:29 PM]: Process name: C:\Capturer\bin\remote_eye_k4a

Ok so it seems to listen. Is there a way to set a more verbose mode (like debug?) Is ti an issue if the sensors have the private network but also a domain network? Either by cable or Wifi,?

No connection shows in the RabbitMQ connection section. Only the volcap connection is shown by the broker. The only time I see the sensors in RabbitMQ conenctions, is when they are started by app with correct IP for broker. How does the service knows the IP of the broker?

[ankarako]

• Is there a way to set a more verbose mode (like debug?)
  • Unfortunately not
• How does the service knows the IP of the broker?
  • volcap sends a broadcast message at a port the service listens to. When the service listens to the broadcast message it opens the remote_eyes.

When you open volcap, the service does not log anything?

[cebdionysos]

No
This is the issue. The sensors are not being contacted by the broadcast from Volcap. This is what I am trying to figure out.
What port UDP/TCP the remote eye is listening on again?

[ankarako]

The service listens to port 11234

[cebdionysos]

To be continued next week. Let me know if you think of anything.

[cebdionysos]

I am back. Been busy.
Did some new test.
So I made sure only private network would be used. So no other network (ie Domain network)
Using one sensor unit for troubleshooting. I monitor the remote_eye_service log to see if it is listening. netstat shows it is on UDP 11234.

If I use portqry from volcalp workstation to check udp port 11234 on sensor it reports listening or filtered, but the log of the remote eye service shows a connection.

Then I restart the service to see again it is freshly listening. Then I open rabbitMQ management page on volcap and start Volcap application. Volcap connects to the broker but doesn't seem to connect to the sensor as nothink shows in Volcap app and neither in the remote eye service log.

1- I need to understand at what step it fails exactly. How does the the volcap app will contact the sensor on port UDP 11234?
2- How is the broker involved in this communication?
3- If the sensor is lstening on udp 11234 and you start Volcap app. What is the exact chain of communication. Fromm volcap to ... and ... to ...

Both 5672 and 15672 TCP ports are set for inbond and outbound on also both Volcap and Sensor. Right?

Thanx

Seb

Seb

[ankarako]

Hi Seb,

When volcap starts it queries the system for the network adaptor (By default it uses the first one from the list returned by the system, as we assume that the workstation has only one network adaptor).
Then it, sends a broadcast message on UDP port 11234 (as you very correctly noticed).
The remote_eye_service listens on UDP 11234, and when a message arrives, it opens the remote_eye (i.e., if the message is "open"). The broker is not involved at all for this kind of communication. This message carries some additional information, such as the IP of the system that hosts the broker.

So from now on, the broker starts to get involved.
When the remote_eye opens, it sends a "handshake" message to volcap through the broker. This message contains some info such as the device name etc.
Then volcap has all the information it needs to connect to the remote_eye and everything happens.

So, in my point of view, probably you may have a "problem" with your network adaptors. We had such experiences in the past, when the system had additional virtual ethernet adaptors, usually prefixed with "v" on Windows, but we have fixed that in the latest version. However, something could be wrong with the way volcap get's the list of network adaptors in the system.

Antonis

[cebdionysos]

From Volcap workstation I open a command prompt and use portqry to connect to remote_eye_service
c:\PortQryV2>PortQry.exe -n 192.168.1.4 -p udp -e 11234

Querying target system called:

192.168.1.4

Attempting to resolve IP address to a name...

Failed to resolve IP address to name

querying...

UDP port 11234 (unknown service): LISTENING or FILTERED

The remote_eye_service log shows :

2022-09-28 2:38:16 PM]: Listener started listening...
[2022-09-28 2:38:16 PM]: Process name: C:\Capturer\bin\remote_eye_k4a
[2022-09-28 3:44:52 PM]: Received broadcast message: PortQry Test Message
[2022-09-28 3:44:52 PM]: Incorrect command line arguments.
[2022-09-28 3:44:52 PM]: Process start info:
[2022-09-28 3:44:52 PM]: Application path:C:\Capturer\bin\remote_eye_k4a
[2022-09-28 3:44:52 PM]: Application cli:

This means that the Volcap workstation can contact the service (but manually with portqry)

When I start volcap,

2022-09-28 2:38:16 PM]: Listener started listening...
[2022-09-28 2:38:16 PM]: Process name: C:\Capturer\bin\remote_eye_k4a

If I do a netstat on volcap workstation after launching the app :

Connecting to broker: { amqp://volumetric:[email protected]:5672}
Exception: send_to: An attempt was made to access a socket in a way forbidden by its access permissions

GL_VENDOR : NVIDIA Corporation
GL_RENDERER : NVIDIA GeForce RTX 3090/PCIe/SSE2
GL_VERSION : 3.2.0 NVIDIA 516.59
GLSL_VERSION : 1.50 NVIDIA via Cg compiler

DDRenderInterfaceCoreGL initializing ...

DDRenderInterfaceCoreGL::setupShaderPrograms()
DDRenderInterfaceCoreGL::setupVertexBuffers()
DDRenderInterfaceCoreGL ready!

nothing shows in this log.

netstat -o

c:\PortQryV2>netstat -o

Active Connections

Proto Local Address Foreign Address State PID
TCP 127.0.0.1:4369 VOLCAP-MASTER:49671 ESTABLISHED 6124
TCP 127.0.0.1:49671 VOLCAP-MASTER:4369 ESTABLISHED 5616
TCP 127.0.0.1:53840 VOLCAP-MASTER:53841 ESTABLISHED 2968
TCP 127.0.0.1:53841 VOLCAP-MASTER:53840 ESTABLISHED 2968
TCP 127.0.0.1:53842 VOLCAP-MASTER:53843 ESTABLISHED 2088
TCP 127.0.0.1:53843 VOLCAP-MASTER:53842 ESTABLISHED 2088
TCP 127.0.0.1:63673 VOLCAP-MASTER:4369 TIME_WAIT 0
TCP 127.0.0.1:63676 VOLCAP-MASTER:4369 TIME_WAIT 0
TCP 192.168.1.1:4369 VOLCAP-MASTER:63671 TIME_WAIT 0
TCP 192.168.1.1:4369 VOLCAP-MASTER:63674 TIME_WAIT 0
TCP 192.168.1.1:5672 VOLCAP-MASTER:57455 ESTABLISHED 5616
TCP 192.168.1.1:5672 VOLCAP-MASTER:57456 ESTABLISHED 5616
TCP 192.168.1.1:5672 VOLCAP-MASTER:57457 ESTABLISHED 5616
TCP 192.168.1.1:5672 VOLCAP-MASTER:57458 ESTABLISHED 5616
TCP 192.168.1.1:57455 VOLCAP-MASTER:5672 ESTABLISHED 12492
TCP 192.168.1.1:57456 VOLCAP-MASTER:5672 ESTABLISHED 12492
TCP 192.168.1.1:57457 VOLCAP-MASTER:5672 ESTABLISHED 12492
TCP 192.168.1.1:57458 VOLCAP-MASTER:5672 ESTABLISHED 12492

It seems like toe broadcast fails to reach the sensor service. Any idea for a next step?

Seb (I will continue tomorrow)

[ankarako]

Thanks for this report.

Well, we have to find from which network adaptor volcap sends the broadcast message.
On Windows, if you go to Control Panel -> Network and Internet -> Network Connections, how many network adaptors do you see?