High latency due to python implementation

[cedricpradalier]

Just for information, because python is sometimes introducing huge delay (like 4-5 seconds!), I was investigating another option to get the camera video stream.

Using gstreamer and gscam seems to work but uses a lot of CPU if it does the decoding itself. The patch below to gscam allows publishing the jpeg stream, same as the axis_driver.
ros-drivers/gscam#11

This looks like a good alternative on the kingfisher, although I haven't found yet how to tell gstreamer to use additional parameter in the get request (like resolution, fps, image size).

[mikepurvis]

Thanks for doing this investigation—performance has been one of the many reasons this driver hasn't received the love it needs from us. If gscam works well for the video portion, then this can evolve into just a stub for managing the PTZ, and the rest of it can be put out of its misery.

[cedricpradalier]

Hi,

Tried the gstreamer-based axis acquisition on the kingfisher today.
The souphttpsrc is working fine, with very little delay, but it uses an
insane amount of CPU, which seems proportional to the file size. With
compression at 0, it uses 20% of a CPU, compression at 10, 8%, compression
at 30, 3-4%. For comparison, the python axis driver using curl, uses 3-4%
at compression 10.

Next, I'll try the neonhttpsrc and we'll see if this gets a bit better.

Regards

On Wed, Nov 27, 2013 at 1:25 AM, Mike Purvis [email protected]:

Thanks for doing this investigation—performance has been one of the many
reasons this driver hasn't received the love it needs from us. If gscam
works well for the video portion, then this can evolve into just a stub for
managing the PTZ, and the rest of it can be put out of its misery.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-29350360
.

Cedric Pradalier

[cedricpradalier]

Hi,

Tested today on the boat with neonhttpsrc (had to compile it from source in
the gstreamer-plugin-bad package). The performance seems has good as the
python script in terms of CPU load (4% CPU load at compression 10), and
much better in terms of latency. We'll test it on the lake at the next
opportunity.

The configuration file is available on
https://github.com/cedricpradalier/axis_camera/blob/master/axis_boot.launch(replace
neonhttpsrc with souphttpsrc to test with the default http
gstreamer source).

HTH

On Mon, Dec 2, 2013 at 11:24 PM, Cedric Pradalier <
[email protected]> wrote:

Hi,

Tried the gstreamer-based axis acquisition on the kingfisher today.
The souphttpsrc is working fine, with very little delay, but it uses an
insane amount of CPU, which seems proportional to the file size. With
compression at 0, it uses 20% of a CPU, compression at 10, 8%, compression
at 30, 3-4%. For comparison, the python axis driver using curl, uses 3-4%
at compression 10.

Next, I'll try the neonhttpsrc and we'll see if this gets a bit better.

Regards

On Wed, Nov 27, 2013 at 1:25 AM, Mike Purvis [email protected]:

Thanks for doing this investigation—performance has been one of the many
reasons this driver hasn't received the love it needs from us. If gscam
works well for the video portion, then this can evolve into just a stub for
managing the PTZ, and the rest of it can be put out of its misery.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-29350360
.

Cedric Pradalier

Cedric Pradalier

[cedricpradalier]

Hi Mike,

Further tests with optimizing the Axis performances. Now that we have
gstreamer taking care of capturing the data and publishing them as jpegs, I
wanted to be able to use the image on-board as well. Problem: decompression
and republish with the standard image_transport tools takes 30% of a CPU
(this is not too bad, gstreamer takes 85%). However, a bit of investigation
and I found out that the jpeg arrive in color (YCbCr) and are converted to
RGB by the republish plugin. Then the first thing I do with them is convert
them to grayscale for image processing. Both convertions are taking a huge
time, but they should not be since the Y component of YCbCr is the
grayscale value.

The imdecode function of opencv does support grayscale decoding directly
(somewhat, it does not use the fact that libjpeg can do direct grayscale
decoding if I remember correctly, but does YCbCr2Gray afterward), but this
is not yet exported in the image_transport plugin. I've raised an issue (
ros-perception/image_transport_plugins#8) but
I've also added a simple node which does that to my axis_camera fork (
https://github.com/cedricpradalier/axis_camera/blob/master/src/republish.cpp).
This is unlikely to get merged because it brings in dependencies on
opencv2/cv_bridge...

On my laptop this save 60% cpu for the decoding. I'll try to check tomorrow
how it behaves on the boat.

Just for information...

On Tue, Dec 3, 2013 at 5:52 PM, Cedric Pradalier <[email protected]

wrote:

Hi,

Tested today on the boat with neonhttpsrc (had to compile it from source
in the gstreamer-plugin-bad package). The performance seems has good as the
python script in terms of CPU load (4% CPU load at compression 10), and
much better in terms of latency. We'll test it on the lake at the next
opportunity.

The configuration file is available on
https://github.com/cedricpradalier/axis_camera/blob/master/axis_boot.launch(replace neonhttpsrc with souphttpsrc to test with the default http
gstreamer source).

HTH

On Mon, Dec 2, 2013 at 11:24 PM, Cedric Pradalier <
[email protected]> wrote:

Hi,

Tried the gstreamer-based axis acquisition on the kingfisher today.
The souphttpsrc is working fine, with very little delay, but it uses an
insane amount of CPU, which seems proportional to the file size. With
compression at 0, it uses 20% of a CPU, compression at 10, 8%, compression
at 30, 3-4%. For comparison, the python axis driver using curl, uses 3-4%
at compression 10.

Next, I'll try the neonhttpsrc and we'll see if this gets a bit better.

Regards

On Wed, Nov 27, 2013 at 1:25 AM, Mike Purvis [email protected]:

Thanks for doing this investigation—performance has been one of the many
reasons this driver hasn't received the love it needs from us. If gscam
works well for the video portion, then this can evolve into just a stub for
managing the PTZ, and the rest of it can be put out of its misery.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-29350360
.

Cedric Pradalier

Cedric Pradalier

Cedric Pradalier

[cedricpradalier]

Hi Mike,

Just for the record, I've modified the command message (and handling) in my
version of the axis driver, to add a field specifying what part of the
command is relevant. This allows controlling the pan,tilt without touching
the exposure or zoom. This also allows implementing exposure control,
independently of the pan-tilt control. Good for tracking objects on the
shore while keeping a good exposure balance.

We've tested the gray-level republish on the boat today and it cuts the
decoding cost by 50%, from 25% to 13% CPU load.

I hope that helps.

On Sun, Dec 8, 2013 at 11:22 PM, Cedric Pradalier <
[email protected]> wrote:

Hi Mike,

Further tests with optimizing the Axis performances. Now that we have
gstreamer taking care of capturing the data and publishing them as jpegs, I
wanted to be able to use the image on-board as well. Problem: decompression
and republish with the standard image_transport tools takes 30% of a CPU
(this is not too bad, gstreamer takes 85%). However, a bit of investigation
and I found out that the jpeg arrive in color (YCbCr) and are converted to
RGB by the republish plugin. Then the first thing I do with them is convert
them to grayscale for image processing. Both convertions are taking a huge
time, but they should not be since the Y component of YCbCr is the
grayscale value.

The imdecode function of opencv does support grayscale decoding directly
(somewhat, it does not use the fact that libjpeg can do direct grayscale
decoding if I remember correctly, but does YCbCr2Gray afterward), but this
is not yet exported in the image_transport plugin. I've raised an issue (
ros-perception/image_transport_plugins#8) but
I've also added a simple node which does that to my axis_camera fork (
https://github.com/cedricpradalier/axis_camera/blob/master/src/republish.cpp).
This is unlikely to get merged because it brings in dependencies on
opencv2/cv_bridge...

On my laptop this save 60% cpu for the decoding. I'll try to check
tomorrow how it behaves on the boat.

Just for information...

On Tue, Dec 3, 2013 at 5:52 PM, Cedric Pradalier <
[email protected]> wrote:

Hi,

Tested today on the boat with neonhttpsrc (had to compile it from source
in the gstreamer-plugin-bad package). The performance seems has good as the
python script in terms of CPU load (4% CPU load at compression 10), and
much better in terms of latency. We'll test it on the lake at the next
opportunity.

The configuration file is available on
https://github.com/cedricpradalier/axis_camera/blob/master/axis_boot.launch(replace neonhttpsrc with souphttpsrc to test with the default http
gstreamer source).

HTH

On Mon, Dec 2, 2013 at 11:24 PM, Cedric Pradalier <
[email protected]> wrote:

Hi,

Tried the gstreamer-based axis acquisition on the kingfisher today.
The souphttpsrc is working fine, with very little delay, but it uses an
insane amount of CPU, which seems proportional to the file size. With
compression at 0, it uses 20% of a CPU, compression at 10, 8%, compression
at 30, 3-4%. For comparison, the python axis driver using curl, uses 3-4%
at compression 10.

Next, I'll try the neonhttpsrc and we'll see if this gets a bit better.

Regards

On Wed, Nov 27, 2013 at 1:25 AM, Mike Purvis [email protected]:

Thanks for doing this investigation—performance has been one of the
many reasons this driver hasn't received the love it needs from us. If
gscam works well for the video portion, then this can evolve into just a
stub for managing the PTZ, and the rest of it can be put out of its misery.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/26#issuecomment-29350360
.

Cedric Pradalier

Cedric Pradalier

Cedric Pradalier

Cedric Pradalier

[mikepurvis]

Looks like the next step here is to just implement a minimal axis-specific C++ framegrabber which is able to be used as both a node and nodelet. I think the ideal would be if axis_camera's driver could advertise separate topics for raw, compressed, and greyscale, and (like other drivers) only produce them when a subscription exists:

• compressed would pass through the frames directly from the camera's http feed.
• raw would use libjpeg to decode the frames and pass them through as bitmaps.
• greyscale would do similar, but use the jpeg_transform_info::force_grayscale option to decode only the Y channel.