Video Streaming framerate infinity

[frapit]

Hi,

the video stream in some bag files appears to be broken, I think the calculation of the frame rate in

bag-database/src/main/java/com/github/swrirobotics/bags/BagService.java

Line 371 in 63196f2

myFrameRate = 1.0 / avgPeriodS;

may need some additional checking.

bagdb_1     | 2021-11-25 08:54:09.273 [mvcTaskExecutor-18] INFO  c.g.swrirobotics.bags.BagService - Starting video stream.
bagdb_1     | 2021-11-25 08:54:09.328 [mvcTaskExecutor-18] DEBUG c.g.swrirobotics.bags.BagService - Image format: bgr24 / 1536x1536 / 0.0s / Infinity Hz
bagdb_1     | 2021-11-25 08:54:09.331 [mvcTaskExecutor-18] INFO  c.g.swrirobotics.bags.BagService - Beginning to stream image data to ffmpeg.
bagdb_1     | 2021-11-25 08:54:09.331 [Thread-127] DEBUG c.g.swrirobotics.bags.BagService - Piping data from ffmpeg to the client.
bagdb_1     | 2021-11-25 08:54:09.398 [Thread-127] DEBUG c.g.swrirobotics.bags.BagService - Finished processing output from ffmpeg.
bagdb_1     | 2021-11-25 08:54:09.398 [mvcTaskExecutor-18] ERROR c.g.swrirobotics.bags.BagService - Error encoding video:
bagdb_1     | java.io.IOException: Broken pipe
bagdb_1     | 	at java.base/java.io.FileOutputStream.writeBytes(Native Method)
bagdb_1     | 	at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
bagdb_1     | 	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
bagdb_1     | 	at java.base/java.io.FilterOutputStream.write(FilterOutputStream.java:108)
bagdb_1     | 	at org.apache.commons.io.IOUtils.write(IOUtils.java:2987)
bagdb_1     | 	at com.github.swrirobotics.bags.BagService$FfmpegImageHandler.process(BagService.java:535)
bagdb_1     | 	at com.github.swrirobotics.bags.reader.BagFile.forMessagesOnTopic(BagFile.java:395)
bagdb_1     | 	at com.github.swrirobotics.bags.BagService.writeVideoStream(BagService.java:806)
bagdb_1     | 	at jdk.internal.reflect.GeneratedMethodAccessor479.invoke(Unknown Source)
bagdb_1     | 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
bagdb_1     | 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
bagdb_1     | 	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:343)
bagdb_1     | 	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
bagdb_1     | 	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:749)
bagdb_1     | 	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
bagdb_1     | 	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
bagdb_1     | 	at com.github.swrirobotics.bags.BagService$$EnhancerBySpringCGLIB$$93f073e2.writeVideoStream(<generated>)
bagdb_1     | 	at com.github.swrirobotics.bags.BagController.lambda$getVideo$0(BagController.java:189)
bagdb_1     | 	at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:110)
bagdb_1     | 	at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:97)
bagdb_1     | 	at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:326)
bagdb_1     | 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
bagdb_1     | 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
bagdb_1     | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
bagdb_1     | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
bagdb_1     | 	at java.base/java.lang.Thread.run(Thread.java:834)
bagdb_1     | 2021-11-25 08:54:09.399 [mvcTaskExecutor-18] ERROR c.g.swrirobotics.bags.BagService - ffmpeg output:
bagdb_1     | [rawvideo demuxer @ 0x5652427c55c0] Unable to parse option value "Infinity" as video rate
bagdb_1     | [rawvideo demuxer @ 0x5652427c55c0] Error setting option framerate to value Infinity.
bagdb_1     | pipe:0: Invalid argument
bagdb_1     | 2021-11-25 08:54:09.399 [mvcTaskExecutor-18] INFO  c.g.swrirobotics.bags.BagService - Done streaming video.

[pjreed]

Hmm, good catch. Looking at the code, while that's possible, I think it would be pretty weird for that situation to happen. I think you would need to have a bag file with an image topic that has at least two frames, but every frame's header would have to have an identical timestamp, causing it to think that that the average period between frames is 0. Do you know if that's the case here?

If a topic doesn't have reasonable timestamps on its image messages, it might not be possible to determine the frame rate for the topic, but I suppose we could just make something up or ask the user for a default frame rate.

Edit: I also think it'd be possible to calculate the frame rate based on the time messages were recorded in the bag file, although it'll take a little work on the bag reader library to make that accessible at this place in the code.

[frapit]

Thanks for the quick answer. Yes you are right, I just checked some of the bags for which the video stream is broken and they all have all timestamps zero on the image topics. That probably causes the issue. I am wondering why rtq tools, etc. have no problem with such messages. Anyway, I think a reasonable default framerate in that case should be sufficient.