Record participant audio and video at the same time within a trial

[pelintanberg]

Hi there,
I am programming a production study where I have to record video and audio for each trial. I need to make sure that participants are saying each word aloud when instructed, but also that they are not mouthing the words on silent trials. I see that I can use html-audio-response and html-video-response to record audio and video separately. Is there a way to combine these?

I also saw that there is a record-video extension but I could not successfully integrate that to the html-audio-response plugin. Any advice or directions are much appreciated. I am using jsPsych 7.3.

Thanks!

[jodeleeuw]

Hi @pelinTan,

There's a parameter in the initialize-camera plugin called include_audio. Setting this to true should allow you to record video with audio using the html-video-response plugin.

const init_camera = {
	type: jsPsychInitializeCamera,
	include_audio: true
}

[pelintanberg]

Thank you. This works great but now I have a question about accessing the saved video+audio file. I am using html-video-response plugin. For my output, I get opus;base64 in the response column. I tried to save video_url too but that link does not work either.

When I previously used html-audio-response, I would get a base64 string, which seems to not happen with html-video-response. Any thoughts? Below is part of my code:

`var init_camera = {
type: jsPsychInitializeCamera,
include_audio: true
}

    var init_microphone = {
        type: jsPsychInitializeMicrophone
    }

    var encoding_word = {
        timeline: [{
            type: jsPsychHtmlVideoResponse,
            stimulus: function(){
                var html = `<div style="font-size:40px;">` + jsPsych.timelineVariable('word1') + `</p>`;
            return html;
            },
            //choices: "NO_KEYS",
            //css_classes: jsPsych.timelineVariable('color'),
            stimulus_duration: DURATIONS.ITEM,
            show_done_button: false,
            recording_duration: DURATIONS.ITEM,
            save_video_url: true,
            //response_ends_trial: false,
            // store word
            data: {trial_kind: "encoding", word1: jsPsych.timelineVariable('word1')},
        }]
    };`

[isagarbisch]

Hi all,
I encountered the same issue with the jsPsychVideoKeyboardResponse plugin. While I was able to record only the video, setting the parameter include_audio: true resulted in opus;base64 in the response column, as it did for @pelinTan.

var intro = {
        type: jsPsychVideoKeyboardResponse,
        stimulus: [
            'videos/fam/Intro_Party.mp4'
        ],
        choices: " ",
        width: widths_desktop,
        trial_ends_after_video: false,
        extensions: [
                    {type: jsPsychExtensionRecordVideo}
                ]
    };

[kschuler]

Does anyone have a solution to this? Both the record-video extension and the html-video-response plugin have this issue. You can't access the base64 data, it just returns the strings: "opus;base64" or 'whatevermimetypeyouspecified;base64".

[kschuler]

I ended up doing this temporary fix, since i need this working today. I am storing the video_url from the previous html-video-response trial, and then using an asynchronous call function to go back and retrieve that; then create the base64Data from that blob.

Sharing in case it helps someone else, but be warned: it's not exactly optimized!

  var init_camera = {
        timeline: [
            {
                type: jsPsychInitializeCamera, 
                include_audio: true, 
                mime_type: 'video/mp4; codecs="avc1.42E01E, mp4a.40.2"'
            }
        ]
    }; 

    var saveRecording = {
        type: jsPsychCallFunction, 
        async: true, 
        func: function(done){
            const video_url = jsPsych.data.get().last(1).values()[0].video_url;
            const trial_index = jsPsych.data.get().last(1).values()[0].trial_index;
            var base64Data = '...';

            fetch(video_url)
                .then(response => {return response.blob()})
                .then(blob => {
                    const reader = new FileReader();
                    reader.onloadend = function() {
                        if (reader.result) {
                            if (typeof reader.result === 'string' && reader.result.startsWith('data:')) {
                                base64Data = reader.result.split(',')[1];  // Extract the base64 string

                                // HERE I SEND TO MY S3 BUCKET
                                const recording = sendToMys3(base64Data, trial_index)
                                done(recording)
                            } else {
                                console.error('Unexpected FileReader result format:', reader.result);
                            }
                        } else {
                            console.error("FileReader result is empty.");
                        }
                    };
                    reader.readAsDataURL(blob);  
                    })
                    .catch(error => {
                    console.error('Error fetching or processing the Blob URL:', error);
                    });
    
        }
    }