Segmentation fault when video size is incorrect

[kojix2]

Hi! GR developers.

Summary

We found that segmentation faults occur when the video size is not specified, or when the video size is not specified correctly.
This is related to x264 and does not occur when using the official GR binaries.
red-data-tools/GR.rb#46

Step to reproduce

red-data-tools distributes a GR package for Linux.
I used this package to install GR . Then set the following environment variables

GKS_WSTYPE=100
GKS_VIDEO_OPTS='361x240@25@2x' or unset GKS_VIDEO_OPTS

Use beginprint() and endprint() to output the video. For example, in the case of Ruby

require 'gr/plot'

GR.beginprint('test.mp4')
GR.plot([1, 2, 3], [4, 5, 7])
GR.endprint

This will cause a segmentation fault.

According to @kou 's research, x264 does not support cases where the video size is not a multiple of 4, so avcode_open2() will fail. (Similarly, it seems to fail if the video size is not specified.) And movie->fmt_ctx->pb will be null. Even in that case, av_write_trailer() will be called and a segmentation fault will occur.

red-data-tools/GR.rb#46 (comment)

This does not happen when using the official binaries distributed in sciapp/gr. The reason for this was thought to be that libopenh264 was used instead of x264.

Thank you.

[danielkaiser]

Hi @kojix2!

Thank you and @kou for reporting the issue and all the research on the problem. This should be fixed in 5c43fd1.
In the new version av_write_trailer is only called when movie->frame has been initialized, which makes sure that the call to avcodec_open2 during initialization was successful and therefore should avoid the segmentation fault when libav tries to write the trailer of a video file that might not be open.

Additionally the video resolution is set to the next larger width/height that is a multiple of 4 if the video is H.264 encoded, so that video creation should no longer fail due to a invalid resolution.

Could you check if this solves your problem?

[kou]

Thanks!
I confirmed that the problem is fixed by the commit.
I think that we can close this issue.

[kojix2]

Thank you @danielkaiser and @kou. I'll close the issue.