Realsense synchronisation

[jackjansen]

There are various issues to be addressed with Realsense inter-camera sync:

• Cameraconfig needs to specify which camera is master (in stead of just picking the first one)
• Cameraconfig must specify the sync mode for slave cameras (see below)
• We need to experiment with external sync
• We need to implement metadata that reports the timestamps of the RGB and D frames used to create the point cloud.

The last is needed for the Tampere dataset capture: if need to sync to an external sync signal, if possible. The signal they have there is a 25Hz square wave at 5V, with the raising edge signalling trigger.

I think we could use master=external (in stead of serial number) for external sync mode.

Slavesyncmode can just be a number (see the librealsense docs).

I think we can build a "good enough" sync generator with an Arduino or esp32 or something like that. I don't think we need a crystal oscillator, but if we do the esp32 has a way to add an external crystal (but this may not be available on all boards).

The last point, timestamp metadata, is essential: at the moment we have no clue as to which frames were used to create the point cloud. And the Realsense documentation states (upon careful reading) that in many of the synchronisation modes the Depth cameras are synced, but the color cameras are essentially free-running independently. In other words: the XYZ of the points would be correct but the RGB could be off by as much as half a frame duration. That will ruin any quality measurement later.

[jackjansen]

https://support.intelrealsense.com/hc/en-us/community/posts/11336312787091-I-can-t-find-the-white-paper-on-external-synchronization-of-RealSense-depth-cameras

[jackjansen]

There is also a lot of interesting information in IntelRealSense/librealsense#10926 (comment)

For one thing: genlock sync may not be what we want, because it only syncs the depth camera, not the color camera.
So we may want to experiment with using "full slave" (which should sync both cameras).

But this will let the capture run freely if no sync signal is available, so we should test in software that we are actually in sync (check that color and depth capture timestamps are the same, and that the delta to the previous capture matches what we expect).

And we may have to reshape the pulse.

[jackjansen]

Here's the signal that comes out of the Tampere sync generator:

Appears to be 4V PP with 33% duty cycle.

[troeggla]

So, I wrote a little ESP32 program which triggers an interrupt on a rising edge to count the number of milliseconds since the interrupt was last triggered.

I am seeing a nice and stable pulse length of 33ms for 30FPS, 40ms for 25FPS and 66ms for 15FPS. I tested this with both the D455 and the D415, with direct connection and through the RC network. So my assumption is just that something with our oscilloscope is bad and the sync signal is fine.

[jackjansen]

@troeggla Very nice! Can you put that program in a repo somewhere? cwi_test maybe, which also has the graycode Arduino thingy already.

[troeggla]

I put it here: https://github.com/cwi-dis/cwipc_test/tree/master/detect_sync_signal

[jackjansen]

For future reference: when I tried to look at the realsense sync signal I got all sorts of different things depending on which oscilloscope I used, but none of them were what I expected. That's why Tom created the device mentioned above.

But I overlooked the fact that digital oscilloscopes sample the signal, and if you're increasing the time base the sample interval will also go up (unlike with analog scopes, which will simply have the beam move slower from left to right):

I think I misunderstand DSOs. If I read https://www.tek.com/en/documents/primer/oscilloscope-systems-and-controls then serious digital scopes have different sampling modes, some of which are specifically meant to forestall this disappearing of short signals to happen (figure 23). All the scopes I tried were relative cheap consumer gadgets.

[jackjansen]

Found an archived copy of the Intel white paper on genlocked sync! (MartyG posted a reference to it in a realsense issue)

External.Synchronization.pdf

[jackjansen]

Initial experiments with genlock syncing have not been very successful. Sometimes it appears to work, for some cameras, but not always. But maybe the notes in the document referenced above, that you should set the camera frame rate to at least twice the real framerate you want, need to be taken into account.