Godot MC Slides (Video and Audio Sync)

[bernarma]

Use Case: Play 2 bits of Media - Video & Sound - at the same time (side-note - the audio is separate to the video and needs to be due to localisation and other reasons).

I have a slide_player in MPF that plays a slide in Godot. That slide (upon activation) will require specific audio to play (that is not embedded in the video that autoplays on the slide). At the moment I am using the event slide_SLIDE_NAME_IN_GODOT_active to trigger the sound_player to play the audio. Although this technique works, it is quite fragile in that the name of the slide is used in the event name and a simple refactor will break this, also, if I try to reuse the same slide to play different videos that require different audio, trying to use the same _active event doesn't provide the additional context I need to play the correct sound.

Here is the YAML (I am typing this out from memory):

slide_player:
  show_my_slide: generic_slide_to_play

sound_player:
  slide_generic_slide_to_play_active: my_audio_to_play

Alternatives I have considered include:

Alternative 1. Playing audio directly in the slide which I have done by creating my own MPF_AudioPlayer class that allows me to drop sounds into the Scene tree (a bodge in my opinion as I have had to "mock" the concept of visibility to work with the MPF base class) - but I feel I am somewhat bypassing the built-in sound_player capability from MPF.

Alternative 2. Having the slide upon activation, sending its own event back to MPF that uniquely identifies the event I need to play the correct audio. This kind of feels like a hack as well as I am basically saying from MPF, play this video, then Godot is saying, OK, I am about to play the video back to MPF, and then MPF receives the event, and tells Godot, can you play this audio clip for me... It feels like there is a lot of event/message chatter with this approach.

Alternative 3. Duplicate the slide to not be reusable which I don't particularly like as having come from a software design background, duplicating the same thing over and over again appears like bad design.

I am keen to hear other peoples ideas about the problem and nice ways to solve it that I haven't considered.

[avanwinkle]

My gut reaction is to make a custom slide with a video+audio playback method that accepts a video name and an audio file name. You could then "play" this slide to add it to the stack and use the slide's active event to trigger an MPF event that posts action: method along with kwargs for the video name and audio file.

Then the slide's custom method would load the video file and audio file and play both, which would be as simultaneous as you could get.

In your mode code:

slide_player:
  mode_foo_started: avsync_slide
  slide_avsync_slide_active:
    avsync_slide:
      action: method
      method: play_av
      tokens:
          video: video1.mp4
          audio: audio1_en.wav

In your custom slide code:

func play_av(settings, kwargs):
  $VideoPlayer.stream = "res://assets/videos/%s" % kwargs.video
  $AudioStreamPlayer.stream = "res://assets/audio/%s" % kwargs.audio

  $VideoPlayer.play()
  $AudioStreamPlayer.play()

Note: those are pseudocode written off the top of my head, actual properties and method names may be different

[bernarma]

I did forget I opened this question up some while ago as I ended up doing exactly as you described.

One of the difficulties I have had with Godot as the media controller is that I "was" forced to use a specific video format that was not compatible with the resources I have... there were also limitations in playing video back from a specific location (which I needed) - i.e. it wasn't supported.

I have managed to extend Godot support for additional video types via gdextensions which has been really helpful, in particular adding support for mp4 encoded videos with resume, but it did come at the expense of coding up the rules for game :-(.

Some of the other difficulties I encountered has been to overcome the sending of values into Godot as parameters that I wanted to be dynamic (resolved at runtime) as opposed to hard coded. At the moment, the tokens are static and the only way I have found I can send in parameters was via the event_player (basically fronting all slide_player invocations with an event_player).

There are other issues that I have come across that I will get around to eventually and may create pull requests for - I have parked the issue for now - support for multiple slide "queues" - but will need to pick it back up once I start adding sound effects into the game. I thought I had a workaround for the multiple queue issue but using multiple displays (which are essentially my queues), I came across a problem with slides not removing from a particular display once it was ready to be removed. I traced the code into the display class and could see that the logic was to remove it (and is empty was allowed) but the display persisted. Anyhow, I'll get around to it at a later stage.

[avanwinkle]

For whatever reason, I cannot figure out how to get GitHub to send me notifications for topics opened in this manner. I can get repo issues filed and PRs opened, but not topics in the Discussion. Sorry for the delay in all my responses.

I'm thrilled you got a GDExtension working for MP4! I've been trying to keep tabs on those but never managed to get one installed or working, but I'm not too smart with those sorts of things and really don't understand what the code does. If you have a working extension and walkthrough, it would be a huge benefit for all us GMC users!!

All variables are stored internally to MPF, but all player and machine variable updates are automatically posted to GMC so if you want to track values there, you've got them. Anything else will have to come from event player (or slide player, you can add tokens there too) at runtime.