feat: volume change event support

[jamsch]

This PR adds a volumechange event that can be used to indicate whether the user is speaking

Addresses issue #35
Closes issue #39

Preview in example app:

[TowhidKashem]

Thanks a lot for doing this, I'd be happy to help test (on ios only, my app doesn't support android atm) if you @ me when it's ready to be tested! :)

[jamsch]

Hey @TowhidKashem, this is now ready to test on version 0.2.24! I've included documentation here to get started.

[TowhidKashem]

Hey @TowhidKashem, this is now ready to test on version 0.2.24! I've included documentation here to get started.

hey sorry for the late reply, I wanted to wait till the weekend to port my app over from react-native-voice first before testing this feature. The migration went very smooth, the well documented types/read me helped a lot! And holy crap the amount of options u exposed in the config is great, love it! I was also able to remove the debounce hack that was needed with the old lib to detect speech end.

As for the volume visualization, it works great for when the user is speaking (input audio). But it seems the lib doesn't support output audio at the moment, is that right? That's something that would be very useful for my use case:

I am building a simulated phone call experience, so when the user is talking I want their speech to be visualized and when they stop talking I send the transcript to the server which creates a response via an LLM and converts it to audio via a TTS service and returns that mp3 to the user which I then play on the phone using expo-av library. It would be great if this part could also be visualized as well as it plays. For that I guess you would need a way to detect volume levels coming from the phone's speaker unlike when the user speaks which u probably detect from the microphone.

Do you think that's something that's possible to support or maybe it's something that should be implemented outside of the lib?

---

Besides this I tested this feature and the lib overall on iOS and I didn't find any issues on a real device (iPhone 13 pro max on iOS 17.6.1)! It felt pretty stable, as I get closer to releasing my app I'll be testing on more devices/simulators and can open issues if I find any.

And just a small suggestion incase it's easy to complement (otherwise ignore this) but it would be nice to be able to configure the duration for how long it takes for the lib to stop listening after speech end. I think now it uses the default system duration of 3 seconds (?) on iPhone but that's a tad too slow for me, maybe like 2.5 or 2 would feel better...

Anyway thanks again for all the work, looks like u put a lot of effort into it and it shows!

[jamsch]

Hey @TowhidKashem, thank you so much for the feedback! To address your points:

• Visualization for audio playback isn't really in scope for this library, however expo-av has an undocumented feature that you may want to try out: Audio.Sound.setOnAudioSampleReceived and here's a getAverageLoudness function that you can implement
• The default non-continuous duration is a hardcoded 3 seconds, but I'm considering adding an API to change this:
  

  expo-speech-recognition/ios/ExpoSpeechRecognizer.swift

  Lines 772 to 781 in 8a60f33

  	self.detectionTimer = Timer.scheduledTimer(
  	withTimeInterval: 3,
  	repeats: false
  	) { [weak self] _ in
  	Task { [weak self] in
  	// Stop listening when the timer fires
  	// This will finish the current task and emit the final result (or a no-speech event)
  	await self?.stopListening()
  	}
  	}

However this should also affect the Android intent setting for EXTRA_SPEECH_INPUT_POSSIBLY_COMPLETE_SILENCE_LENGTH_MILLIS / EXTRA_SPEECH_INPUT_COMPLETE_SILENCE_LENGTH_MILLIS / EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS

We'll also need to add additional timer to match the Android EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS to avoid prematurely cutting the user off before they get a chance to speak.

[TowhidKashem]

• Visualization for audio playback isn't really in scope for this library, however expo-av has an undocumented feature that you may want to try out

That makes sense don't worry about it then! And thanks for letting me know about the undocumented feature in expo-av. I would have never found that otherwise! If that doesn't work I was thinking I might be able to use the audio file as input source in this lib (since u seem to support that):

    ExpoSpeechRecognitionModule.start({
      audioSource: {
        /** Local file URI */
        uri: "file:///path/to/audio.wav",

...and get the sound levels that way somehow. Cause now I send the audio from the server to the RN side as a base64 string which I then play directly in expo-av via loading it as a data uri string. Expo-av I believe still needs to save this file to the file system in the background prior to playing it so I might be able to use that path here in the uri option. And I was guessing might be able to detect volume levels that way? But since you're suggesting the expo-av undocumented feature route I'll start with that first and see how it goes 🤞

We'll also need to add additional timer to match the Android EXTRA_SPEECH_INPUT_MINIMUM_LENGTH_MILLIS to avoid prematurely cutting the user off before they get a chance to speak.

Gotcha, makes sense. yea it's not a deal breaker or anything, I can live with the 3 seconds time, just a nice to have if it can be done on ur end without a hacky patch for the android side. And thanks for showing me the hard coded location, In the meantime I will just use patch package or a expo config plugin to change that value for now 👍

[jamsch]

@TowhidKashem Also a FYI, at one point when expo-audio becomes recommended over expo-av, there's also APIs to sample the volume, here's an example: https://github.com/expo/expo/blob/95dec52592e24b45ed056cc4599640fb6bd97726/apps/native-component-list/src/screens/Audio/expo-audio/JsiAudioBar.tsx#L35-L42

[TowhidKashem]

@TowhidKashem Also a FYI, at one point when expo-audio becomes recommended over expo-av, there's also APIs to sample the volume, here's an example: https://github.com/expo/expo/blob/95dec52592e24b45ed056cc4599640fb6bd97726/apps/native-component-list/src/screens/Audio/expo-audio/JsiAudioBar.tsx#L35-L42

hey @jamsch sorry just seeing your mention now, but this is great news thanks for letting me know! I'll give this a shot once SDK 52 comes out of beta in a week or two. I did try to use the undocumented apis from expo-av you mentioned before but it didn't work out because it seems they only work if you are recording the audio. In my case I don't want to record, just visualize the sound as it happens in the environment. I couldn't find the documentation page for expo-audio like there is for expo-video, I guess they haven't created it yet do you know if this useAudioSampleListener hook can in fact be used to visualize audio levels without recording?

[jamsch]

@TowhidKashem Yep, looks like expo-audio supports sampling playback audio.

Once you use useAudioSampleListener(...), it calls player.setAudioSamplingEnabled(true) on the native Android/iOS side to start sampling audio:

https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/packages/expo-audio/src/ExpoAudio.ts#L38-L52

Which has the following implementations:

Android:
https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/packages/expo-audio/android/src/main/java/expo/modules/audio/AudioPlayer.kt#L49-L70

iOS:
https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/packages/expo-audio/ios/AudioPlayer.swift#L121-L149

[jamsch]

@TowhidKashem By the way, it does seem like you can sample playback audio on expo-av, just looking at the following:

https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/apps/native-component-list/src/screens/Audio/AV/JsiAudioBar.tsx#L52-L64

[TowhidKashem]

@TowhidKashem By the way, it does seem like you can sample playback audio on expo-av, just looking at the following:

https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/apps/native-component-list/src/screens/Audio/AV/JsiAudioBar.tsx#L52-L64

Hmm I just tried it but it gives this error:

TypeError: sound?._updateAudioSampleReceivedCallback is not a function (it is undefined)

I am on SDK ^51.0.36 and expo-av ^14.0.7. I guess the private method isn't accessible to me for some reason...

But I'll def give it a shot with expo-audio and sdk 52 soon, will update here how it goes, thanks again for the heads up!

[jamsch]

@TowhidKashem It looks like you also need to call sound?._updateAudioSampleReceivedCallback(); prior to that:

https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/apps/native-component-list/src/screens/Audio/AV/JsiAudioBar.tsx#L27-L38

[TowhidKashem]

@TowhidKashem It looks like you also need to call sound?._updateAudioSampleReceivedCallback(); prior to that:

https://github.com/expo/expo/blob/afea64c2d81b5dfba9e4579c95c0aaec7539fa63/apps/native-component-list/src/screens/Audio/AV/JsiAudioBar.tsx#L27-L38

With some chat GPT'ing I got it to finally work! Still not sure what the original error was caused by but first I got it to work on an audio file and then from the mic in recording mode. The horizontal bar is moving in response to the sound. The only issue is that the sound is saved afterward. But that's not a huge deal, I can just delete each audio clip afterward. And the recording light next to the camera is also on the whole time but I don't think that's possible to disable anyway since the audio technically is coming form the mic. Anyways glad it's finally working on both ends, thanks for the help!