Communication lost between intervideosink and intervideosrc

[sporteiro]

Hi there, congrats for the project!

After a couple or tests with different inputs and outputs, I just realized there was always some frozen frames, (no matter what the input/output combination was). Digging a bit on the code, I figured out that if I set the timeout to 0 in outputs/output.py like this:

def _video_pipeline_start(self):
    '''
    The standard start to the pipeline string for video.
    It starts with intervideosrc, which accepts video from the source.
    '''
    # The large timeout holds any stuck frame for 24 hours (basically, a very long time)
    # This is optional, but prevents it from going black when it's better to show the last frame.
    #timeout = Gst.SECOND * 60 * 60 * 24
    timeout = 0
    return ('intervideosrc name=intervideosrc timeout=%d ! videoconvert ! videoscale ! '
            'videorate ! capsfilter name=capsfilter ! ' % timeout)

The output stream is always having some black frames (Can be more or less depending on the video caps, but they're always there). Example with this config file:

mixers:
- height: 720
  id: 1
  pattern: 0
  sources: []
  state: PLAYING
  type: mixer
  width: 1280
outputs:
- audio_bitrate: 128000
  container: mpeg
  height: 720
  host: 127.0.1.1
  id: 1
  port: 7000
  source: mixer1
  state: PLAYING
  type: tcp
  width: 1280

Output of:

ffmpeg -i tcp://127.0.1.1:7000 -vf "blackdetect=d=0:pix_th=0.00" -an -f null -

Input #0, mpegts, from 'tcp://127.0.1.1:7000':
  Duration: N/A, start: 3601.793733, bitrate: N/A
  Program 1 
    Stream #0:0[0x41]: Video: h264 (High) (HDMV / 0x564D4448), yuv420p(tv, bt709, progressive), 1280x720 [SAR 1:1 DAR 16:9], 30 fps, 30 tbr, 90k tbn, 60 tbc
    Stream #0:1[0x42](en): Audio: ac3 (AC-3 / 0x332D4341), 48000 Hz, stereo, fltp, 128 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf57.83.100
    Stream #0:0: Video: wrapped_avframe, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc
    Metadata:
      encoder         : Lavc57.107.100 wrapped_avframe
[blackdetect @ 0x55a9d5dbf000] black_start:0.2 black_end:0.233344 black_duration:0.0333444
[blackdetect @ 0x55a9d5dbf000] black_start:2.1 black_end:2.13334 black_duration:0.0333444
[blackdetect @ 0x55a9d5dbf000] black_start:2.33334 black_end:2.36667 black_duration:0.0333222
[blackdetect @ 0x55a9d5dbf000] black_start:2.86667 black_end:2.9 black_duration:0.0333333
[blackdetect @ 0x55a9d5dbf000] black_start:4.6 black_end:4.63334 black_duration:0.0333444
[blackdetect @ 0x55a9d5dbf000] black_start:5.93334 black_end:5.96667 black_duration:0.0333222
[blackdetect @ 0x55a9d5dbf000] black_start:6.9 black_end:6.93334 black_duration:0.0333444
[blackdetect @ 0x55a9d5dbf000] black_start:7 black_end:7.03334 black_duration:0.0333444
[blackdetect @ 0x55a9d5dbf000] black_start:7.8 black_end:7.83334 black_duration:0.0333444
[blackdetect @ 0x55a9d5dbf000] black_start:7.86667 black_end:7.9 black_duration:0.0333333
[blackdetect @ 0x55a9d5dbf000] black_start:8.33334 black_end:8.36667 black_duration:0.0333222
[blackdetect @ 0x55a9d5dbf000] black_start:8.63334 black_end:8.66667 black_duration:0.0333222
[blackdetect @ 0x55a9d5dbf000] black_start:8.76667 black_end:8.8 black_duration:0.0333333
[blackdetect @ 0x55a9d5dbf000] black_start:10.6333 black_end:10.6667 black_duration:0.0333222

I tested separately an example pipleline with Gstreamer like this:

gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=1280,height=720,framerate=30/1 ! videoconvert ! intervideosink name=psink intervideosrc name=psrc timeout=0 ! video/x-raw,width=1280,height=720,framerate=30/1 ! videoconvert ! autovideosink

And, although its not behaving "perfect" (1 or 2 black/hour), it seems to be "loosing" less data than Brave.

Im running my tests in different computers, with/whithout virtual machines and with/without Docker, with minimal differences

Now my question is, Are you aware of this behave? Did you manage to solve or mitigate it?

Best Regards and thank you very much

[moschopsuk]

Hello @sporteiro,

Thanks for the information. It is something we have noticed. Your very detailed analysis does show that over time due to the way inter elements split the stream that it could cause black frames. We using brave this results in overtime the A/V will drift out of sync.

Other people have also found this issue with the inter elements:
http://gstreamer-devel.966125.n4.nabble.com/debugging-a-v-sync-issues-possibly-inter-element-related-td4672079.html
voc/voctomix#58

As for this repo, we have a version that has not been yet merged into here that replaces the intervideo(sink/src) with the interpipe(sink/src) available from here:
https://github.com/RidgeRun/gst-interpipe

Hopefully this can help.

[sporteiro]

Thanks for your reply @moschopsuk,

I will be looking forward to see that version with interpipe, in fact was one of the ideas to possibly solve the issue from my side. Just so you know, later checks with non virtualized and more powerful machine, showed a far better performance, but still black frames from time to time (similar to the basic pipeline on my example).

Thank you very much for your work and Happy New Year!