AVC / H.264 video support

[amacneil]

Currently Foxglove Studio supports rendering image frames using the sensor_msgs/Image and sensor_msgs/CompressedImage. The latter supports PNG or JPEG compression.

Several people have asked for the ability to render video data, most commonly in h264 format. Often, this data is recorded as a separate file alongside a ROS bag, but in theory the NAL units could be wrapped within a ROS or Protobuf message, or stored directly in MCAP.

Several methods to implement this have been discussed, from using native HTML video elements, to decoding the frames in WASM.

Related tickets: https://github.com/foxglove/studio/issues/1069 https://github.com/foxglove/studio/issues/537

[Aposhian]

This would be very useful. ROS Image topics are horribly inefficent, since they are raw image frames without intra-frame or frame-to-frame compression (like H.264). Specifically, being able to subscribe to RTSP streams would be amazing.

[jtbandes]

@Aposhian How do you imagine those streams would be synced with other message data coming from the main data source?

[Aposhian]

I suppose what I was envisioning was primarily supporting the live use case, where you are streaming from a dedicated video source that is assumed to be synced with your primary data source. However, absolute timing information can be obtained from a container protocol like RTSP. While I may be incorrect on this, it seems that RTCP messages can contain NTP timestamps, and RTP packets contain relative timestamps, so that by combining the two, you can obtain an absolute timestamp for each RTP packet. Then, if your RTSP source is synced with the same NTP server as your primary data source, it should be synced. I've primarily used RTSP in my applications, so I'm unsure of what container formats with H.264 data other people would want to visualize in foxglove.

[amacneil]

Related: We should support visualizing h264 directly in MCAP (foxglove/mcap#427)

[lishulincug]

https://github.com/foxglove/studio/issues/3739#issue-1286687657

[lishulincug]

I suppose what I was envisioning was primarily supporting the live use case, where you are streaming from a dedicated video source that is assumed to be synced with your primary data source. However, absolute timing information can be obtained from a container protocol like RTSP. While I may be incorrect on this, it seems that RTCP messages can contain NTP timestamps, and RTP packets contain relative timestamps, so that by combining the two, you can obtain an absolute timestamp for each RTP packet. Then, if your RTSP source is synced with the same NTP server as your primary data source, it should be synced. I've primarily used RTSP in my applications, so I'm unsure of what container formats with H.264 data other people would want to visualize in foxglove.

How to set the rosbag's image into a RTSP

[Aposhian]

Not sure if this is the correct way to go about it, but it might also be good to take note of https://github.com/clydemcqueen/h264_image_transport

[Aposhian]

What is the current thinking on this? Even being able to display H.264 frames packed into ROS messages would be very helpful, since that would be a lot more storage and bandwidth efficient.

[defunctzombie]

The latest on this that I've seen is a community member that worked on an Studio extension to view h264 data that was packed in annex-b format inside mcap messages:

https://github.com/codewithpassion/foxglove-studio-h264-extension

[chfritz]

I'm working on a live connector for Foxglove that uses webrtc (over UDP) for the connection and h264 for video-compression: https://transitiverobotics.com/caps/transitive-robotics/foxglove-webrtc/. It's still early but it's already usable and if anyone is interested I'd love a design partner.

[amacneil]

Very cool!

[Aposhian]

Very interesting! Although I'd still love to see a solution that works for bagged data.

[chfritz]

Thanks, @Aposhian . BTW, the underlying video-streaming capability supports RTSP streams as well and I'll probably enable those in the foxglove-connector as well -- I think you asked about that.
Re. bagged data: do you mean a way to store h264 video in a bag and play it back in foxglove? That makes sense, too. Is the primary motivation for this the storage efficiency, smaller bags? If so, is that because you want to keep more bags on the robot than you currently can due to disk limitation, or are you thinking of an efficiently-stream-to-cloud and store there solution?

[Aposhian]

do you mean a way to store h264 video in a bag and play it back in foxglove?

Yes

Is the primary motivation for this the storage efficiency, smaller bags?

Yes

If so, is that because you want to keep more bags on the robot than you currently can due to disk limitation, or are you thinking of an efficiently-stream-to-cloud and store there solution?

Not necessarily streaming, but yes, copying around, uploading and downloading very large bags is the largest problem.

[emersonknapp]

I remember https://github.com/aws-robotics/kinesisvideo-ros1/blob/master/kinesis_video_msgs/msg/KinesisVideoFrame.msg - this project isn't live any more, and I'm not sure if there's other generic encoded data stream msgs predefined anywhere.

But it seems pretty sensible to me to have a H264.msg specific codec messages, or a more generic like above. Perhaps a project like https://github.com/ros-perception/image_transport_plugins would be a good place for it. It definitely wouldn't get the full benefit of a raw connected h264 data buffer, but would allow for at least storing the data frames in mcap/rosbag gaining the compression benefits of that.

EDIT: I hadn't read the prior discussion closely enough - I see now that https://github.com/clydemcqueen/h264_image_transport/blob/master/h264_msgs/msg/Packet.msg is an attempt at this approach, thanks for the link @Aposhian

[yangxin0]

I think the best way to play h264 in web browser is webrtc. we can implement a signaling server in webbridge server and the sensor node can directly encode frames as RTP packet and send to foxglove ( with some simple extra handshake).

[yangxin0]

Some Products also have more or less remote control needs like my auto sweeper product, so we can send RTP packet to both foxglove and remote endpoint for remote control. This can reduce many IO workloads because of transfering encoded frame and reuse of RTP packet.

[chfritz]

@yangxin0 , agreed. That's the thinking behind the Foxglove webrtc connector I've implemented: https://transitiverobotics.com/caps/transitive-robotics/foxglove-webrtc/. Let me know if you would like to try it.

[Aposhian]

Although WebRTC is not compatible with viewing bags and is only suitable for live view, as far as I can tell. Unless we somehow record the packets and the control signals.

[lucumt]

We have also implemented a RTSP extension https://github.com/foxglove-custom/rtsp-player-extension

[defunctzombie]

Yesterday we announced H.264 support in foxglove. This is available across all of our built-in data-sources (mcap, bag, websocket, data-platform streaming) and plays in-sync with all of your other data.

https://foxglove.dev/blog/announcing-h264-support-in-foxglove

[achim-k]

Do you have any example ros1/2 mcaps with h264 video in them?

At https://mcap.dev you can record webcamera h264 frames into an mcap file

Or I can create my own by loading an mcap or rosbag with image topics and doing something with a converter or user script, or are those only for converting messages live, not for creating a new output mcap/bag file?

Only for converting messages live

[lucasw]

Let me know if this should go somewhere else (e.g. mcap-videoify issues but issues aren't enabled there, or a new discussion).

This was recorded with https://mcap.dev/: demo_2023-11-18_06-06-59.zip - if there is a known good mcap of just foxglove.CompressedImage, or a tool to convert ros1 or ros2 image messages into that format (and looks like it has to be protobuf), or an example of encoding numpy/cv2 image arrays I'd try that out (https://github.com/foxglove/mcap/blob/main/python/examples/protobuf/image_example.py maybe could be adapted for that?).

$ mcap info demo_2023-11-18_06-06-59.mcap 
library:   MCAP web demo                                         
profile:                                                         
messages:  381                                                   
duration:  6.376s                                                
start:     2023-11-18T06:06:37.221-08:00 (1700316397.221000000)  
end:       2023-11-18T06:06:43.597-08:00 (1700316403.597000000)  
compression:
	zstd: [242/242 chunks] [5.69 MiB/5.69 MiB (-0.02%)] [913.89 KiB/sec] 
channels:
	(0) pose           1 msgs              : foxglove.PoseInFrame [protobuf]      
	(1) camera_jpeg  190 msgs (29.80 Hz)   : foxglove.CompressedImage [protobuf]  
	(2) camera_h264  190 msgs (29.80 Hz)   : foxglove.CompressedVideo [protobuf]  
attachments: 0
metadata: 0

(I realize there's already CompressedVideo in the mcap, but the point is having tools to convert other formats into it, and mcap-videoify ignores messages that aren't CompressedImage so no issue there)

$ mcap-videoify demo_2023-11-18_06-06-59.mcap
thread 'main' panicked at .../protobuf-3.3.0/src/reflect/name.rs:21:5:
full name must start with dot: google.protobuf.Timestamp

I hadn't seen openh264 before, maybe that's a much cleaner solution than using ffmpeg and supports all of python/c++/rust.

[chfritz]

I hadn't seen openh264 before, maybe that's a much cleaner solution than using ffmpeg and supports all of python/c++/rust.

Another option is gstreamer which I recommend over ffmpeg, which supports python/c++/rust (+javascript, via node-gtk).

[lucasw]

Another option is gstreamer which I recommend over ffmpeg, which supports python/c++/rust (+javascript, via node-gtk).

Is there example code that maps well to the live ros use-case: one image added to a persistent encoder object and one new encoded array of bytes out (to then put into a CompressedVideo or similar message), and one encoded data packet into a decoder object and one image out that corresponds exactly to the timestamp sent alongside the encoded data (and assuming no b-frames, and have a key frame)?

[chfritz]

That would be a pipeline like this (assuming rgb8 ros images, with WIDTH and HEIGHT filled in accordingly):

appsrc name=src ! video/x-raw,format=RGB,width=WIDTH,height=HEIGHT ! \
 videoconvert ! video/x-raw,format=I420 ! queue ! \
 x264enc tune=zerolatency bitrate=400 ! \
 appsink name=sink

Once started, you'd be getting the appsrc and appsink elements by name in your code, start feeding the source with image buffers, and receiving new-sample signals (events) from the sink.

This is the kind of stuff we are using in the webrtc-video capability that me and Mike have been mentioning to you in the past, and also in our foxglove-webrtc live connector.