The sound is captured back to the microphone

[hdv0112]

Hi BananaHemic,
I have big problem with my voice chat App. When 2 people voice each other by mobile phone. The voice sound of the speaker is recorded back to the microphone, and causing echo. After many times the sound is echoed and resonant with each other, causing loud noises. I've been looking for solutions like AEC(Acoustic Echo Cancellation), Echo Suppression, but it's not work for Unity.
Have you ever had this problem or has any solutions, please let me know. Thanks so much!!!

[BananaHemic]

I also have this issue, and it is definitely something I'd like to solve. I personally would probably just write a script that would listen to the audio output from the MubleAudioPlayer, and build an internal circular float array of previously played audio. Then, after N milliseconds I would pull the most recent microphone data and do a sliding dot product to estimate the latency from audio read -> mic read. Then, if the output value from the sliding dot product is sufficiently high, I'd subtract the past played audio from newly pulled microphone audio buffers.

I haven't written this yet, but I'm quite confident that it would work. If you decide to work on this, I'd be happy to lend a hand in any way I can

[hdv0112]

Thanks for your reply. But, can you describe more clearly? :)
I really want to fix it!

[BananaHemic]

All you need is 1) an array with the audio that was previously played and 2) how long it takes for audio that was played to reach the mic. Then you just subtract the previously played audio from the mic input, offset by the latency from 2).

[rkachach]

Thank you very much for this great project. It's really amazing :)

I'm using the plugin for a VR app I'm working on and I'm having the same issue ... is there a better approach for example to use hardware echo cancelation (for example on Android)?

[sebjf]

@rkachach on Android the Unity Microphone class will provide a stream with the AEC and NS effects enabled (at least on the devices I've tested). Otherwise, you can use the JNI to add them.

Here is a snippet showing how to inspect the audio effects on an AudioRecord when the Microphone is opened in Unity.

  try
  {
      AndroidJavaClass unityPlayer = new AndroidJavaClass("com.unity3d.player.UnityPlayer");
      AndroidJavaObject activity = unityPlayer.GetStatic<AndroidJavaObject>("currentActivity");
      audioManager = activity.Call<AndroidJavaObject>("getSystemService", "audio");
  }
  catch (Exception e)
  {
      Debug.Log(e.ToString());
  }

private void Update()
{
    if(audioManager != null)
    {
        var activeRecordingConfigurations = audioManager.Call<AndroidJavaObject>("getActiveRecordingConfigurations");
        var size = activeRecordingConfigurations.Call<Int32>("size");

        if(size != previousActiveRecordingConfigurationsSize)
        {
            previousActiveRecordingConfigurationsSize = size;
            Debug.Log("Changed Audio Recording Configurations");

            for (int i = 0; i < size; i++)
            {
                var audioRecordingConfiguration = activeRecordingConfigurations.Call<AndroidJavaObject>("get", i);
                var audioSource = audioRecordingConfiguration.Call<Int32>("getAudioSource");
                var audioSessionId = audioRecordingConfiguration.Call<Int32>("getClientAudioSessionId");
                var audioClientEffects = audioRecordingConfiguration.Call<AndroidJavaObject>("getClientEffects");

                Debug.Log($"Audio Recording Configuration {i}");
                Debug.Log($"Audio Source {audioSource}");
                Debug.Log($"Audio Session Id {audioSessionId}");

                var audioClientEffectsSize = audioClientEffects.Call<Int32>("size");
                for (int j = 0; j < audioClientEffectsSize; j++)
                {
                    var audioClientEffectDescriptor = audioClientEffects.Call<AndroidJavaObject>("get", j);
                    var audioClientEffectName = audioClientEffectDescriptor.Get<string>("name");

                    Debug.Log($"Audio Effect {j} {audioClientEffectName}");
                }
            }
        }
    }
}

You could use the JNI similarly to add the effect to Unity's session if it were not added automatically.

However, we have found mixed results. On the Oculus Quest's for example we are finding the inbuilt AEC doesn't work half the time.

[djavadihamid]

@sebjf
Thanks in advance for sharing this.

I would appreciate if you describe more about this.
How can I add aec effect to MumbleClient?
I am new to this topic and I really need this.

Thanks

@BananaHemic
It would be great if you can help and solve this problem. I know you spent a lot of time to work on this and in fact, this is a nice library but I am unable now to use it just because of this little issue.

Thanks a lot

[sebjf]

Hi @djavadihamid, I wouldn't be able to say how to add it to Mumble! I was searching round the topic and came across this the other day, and thought I'd post what I'd found so far since there is very little documentation about this topic in general.

In Android, it is possible to add the AEC effect using just the session ID (that you can get from the above). I haven't tested it but you should be able to create an AndroidJavaClass object with the AcousticEchoCanceler name, then CallStatic the create method and pass in the session id.
Then, regardless of how the mic is opened (e.g. Unity, custom Java lib, OpenSLES, etc), you can add the effect.

Though, on modern Android you may find the effects are already added (it will be pretty obvious from the messages in logcat - the name of the effect will be something like "AcousticEchoCanceler", for example).

However, I'd warn you again the HW AEC may not be a cure all: it is enabled on our app but we still have some echo issues.

[djavadihamid]

However, I'd warn you again the HW AEC may not be a cure all: it is enabled on our app but we still have some echo issues.

Again Thanks for your reply.

I am going to use it for android.
You mean it may not work for all android devices ?

[sebjf]

You mean it may not work for all android devices ?

Some devices don't support HW AEC, but also on those that do it may just not be a very good implementation.

[djavadihamid]

@BananaHemic

So finally I think it should be handled in code as you said.
I really want your help to fix this issue.
Thanks a lot

[BananaHemic]

So finally I think it should be handled in code as you said.
I really want your help to fix this issue.
Thanks a lot

I can point you in the right direction, and maybe look over code that you've written, but unfortunately I just don't have time to fix this for you

[djavadihamid]

I can point you in the right direction, and maybe look over code that you've written, but unfortunately I just don't have time to fix this for you

Oh sad! I am absolutely new to this project
But that is fine
I found your repo a good one and I do whatever I can to fix this little issue
so how can I get in touch with you for doing this?

[BananaHemic]

so how can I get in touch with you for doing this?

If you're serious about doing this, then I'd suggest making a fork of this repo, and asking me questions as you go.

If, on the other hand, you're looking for something that's do-able in a day, then I'd suggest focusing on the HW AEC.

First step would be to pull unity's output audio with timestamps, then place it into a circular buffer. If you care about avoiding memory allocations (like I did when I was making this repo!) then this becomes fairly non-trivial. You might get some inspiration from DecodedAudioBuffer

If this script works, you should then be able to read what the audio output X milliseconds ago. Once you make this script, follow up here with a link.