Extremely slow FPS for depth on Ubuntu due to packet loss when using a second monitor

[RobinVogel]

Overview Description: The refresh rate in Protonect is super low for the depth ( with options -norgb). The refresh rate is completely acceptable with -nodepth. The problem does not arise if I disconnect my second monitor (connected with HDMI). It does not seem to be related to the processing speed of the data but to a loss of packets, since I have in my output:

[Info] [TurboJpegRgbPacketProcessor] avg. time: 14.7637ms -> ~67.7336Hz
[Info] [OpenGLDepthPacketProcessor] avg. time: 10.3735ms -> ~96.3996Hz

which seems pretty acceptable. However, my output gets flooded with messages of this type:

[Info] [DepthPacketStreamParser] 167 packets were lost
[Info] [DepthPacketStreamParser] 45 packets were lost
[Info] [DepthPacketStreamParser] 41 packets were lost
[Info] [DepthPacketStreamParser] 41 packets were lost
[Info] [DepthPacketStreamParser] 140 packets were lost

Which are also there when using --noviewer but obviously disappear when using --nodepth.

The issue seems related to:

• this one ((not all subsequences received / packets were lost) by using multiple kinects #844) but was complaining with low packet loss,
• this one (Packet losses on Windows 10 #588) but on windows,
• this one (Protonect packet loss without rgb stream #783) but on mac,
• this one (Loss of Depth Packets #637) but on windows,

hence why I create a new issue, as none of these were solved nor are related to my case.

Version, Platform, and Hardware Bug Found:

1. git log -1 --oneline

fd64c5d (HEAD -> master, tag: v0.2.1, origin/master, origin/HEAD) Update README.md

2. uname -a

Linux laptop-robin 5.13.0-30-generic #33~20.04.1-Ubuntu SMP Mon Feb 7 14:25:10 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

4. lsusb -t

/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 1: Dev 2, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 1: Dev 2, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 2: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 12M
    |__ Port 2: Dev 3, If 1, Class=Human Interface Device, Driver=usbhid, 12M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 10000M
    |__ Port 2: Dev 4, If 0, Class=Vendor Specific Class, Driver=, 5000M
    |__ Port 2: Dev 4, If 1, Class=Vendor Specific Class, Driver=, 5000M
    |__ Port 2: Dev 4, If 2, Class=Audio, Driver=snd-usb-audio, 5000M
    |__ Port 2: Dev 4, If 3, Class=Audio, Driver=snd-usb-audio, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 480M
    |__ Port 3: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 3: Dev 2, If 1, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 4: Dev 3, If 0, Class=Video, Driver=uvcvideo, 480M
    |__ Port 4: Dev 3, If 1, Class=Video, Driver=uvcvideo, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/2p, 480M

6. lspci -nn

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Root Complex [1022:1630]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Renoir IOMMU [1022:1631]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
00:02.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 0 [1022:1448]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 1 [1022:1449]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 2 [1022:144a]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 3 [1022:144b]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 4 [1022:144c]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 5 [1022:144d]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 6 [1022:144e]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir Device 24: Function 7 [1022:144f]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU116M [GeForce GTX 1660 Ti Mobile] [10de:2191] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU116 High Definition Audio Controller [10de:1aeb] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU116 USB 3.1 Host Controller [10de:1aec] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] [10de:1aed] (rev a1)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
03:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8822CE 802.11ac PCIe Wireless Network Adapter [10ec:c822]
04:00.0 Non-Volatile memory controller [0108]: Micron Technology Inc Device [1344:5404] (rev 03)
05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Renoir [1002:1636] (rev c7)
05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
05:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
05:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
05:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
05:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor [1022:15e2] (rev 01)
05:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)
06:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 81)

Steps to Reproduce:

1. Tried to compile with and without VAAPI, chose without VAAPI as it removed some errors I encountered,
2. Tried to use protonect with cl, gl or cuda. But I understand why it's useless as it's not my depth image processing times that are at fault,
3. LibUSB=3 gives an error "libusb: error [udev_hotplug_event] ignoring udev action change" libfreenect2
4. Crazy story: I could completely correct that bug by just unplugging my second monitor... No idea why though...

[smokhov]

Such things are most often tied to the USB3 controllers, their driver support in the kernel; lesser to USB3 cables (some are really faulty), and finally defective connectors or the Kinect itself. Did you you verify your Kinect works as expected with your cable on another machine with the native SDK samples? You also seem to have 2 types of USB3 controllers -- by AMD and by NVIDIA... did you try both? I also had cases when my USB3 connectors were flimsy in the front, but the ones at the back of the PC worked fine. I also had cases with bad cables.

[RobinVogel]

I do not have access to another machine at the moment but will test that and report back in ~3 weeks from now.
I have only one USB 3.0 port (tried them all, only one functions with the kinect), I don't really know about the distinction between usb port and usb controller, but I guess you deduced that from the lspci output.

I don't think the issue is related to the cables as:

• the HDMI screen works perfectly with or without the kinect,
• the kinect works perfectly without the HDMI screen plugged in.

The result is still mysterious to me, but on my side I will use the kinect without the HDMI screen for now.
But the issue is interesting and I will help if anyone wants to dig further.