Add support for two-way audio for SIP based video doorbell #1253

nanosonde · 2022-01-20T16:39:25Z

Hi!

This PR addresses this issue #928.

It adds support for two-way audio between Homekit and a SIP-based video doorbell.
Those video doorbells based on SIP seem to exist in at least two different flavours:

Video is a completely seperate thing: e.g. MJPEG stream via HTTP only. SIP is only used for Audio.
Video is a H264 stream that is also negotiated as an additional stream (next to Audio) during SIP signaling.

This PR only addresses variant 1). However, adding variant 2) should not be too difficult.

The solution presented here does not involve using audio devices (e.g. alsa loopback) or the like and it does not use an external SIP softphone to handle the SIP call.
It uses the well-known SIP stack from kirm.

The SIP call is a direct SIP call between two peers. No proxy or registrar is involved.
A SIP INVITE is sent with the suggested RTP stream and codec parameters (SDP) and the related SIP response SDP from the video doorbell is parsed.
Based on these two SDPs the two FFmpeg instances are configured accordingly.
As a result the SIP video doorbell sends and receives the audio streams directly to/from the homebridge-camera-ffmpeg plugin (two ffmpeg instances). Full-Duplex.

For now the negotiated codec towards the SIP doorbell is hard-coded to G.711 (u-Law). All SIP devices should support this codec. For example, my doorbell only supports G.711 u-Law and A-law.

At the following to your camera config:

                  "sipConfig": {
                        "to": "sip:[email protected]",
                        "from": "sip:[email protected]"
                    }

A big thank you goes to this project: https://github.com/dgreif/ring
The Ring cameras speak SIP towards the Ring servers. So I learned all the basics from dgreifs solution.

The RTPhelper was inspired by this project: https://github.com/hjdhjd/homebridge-unifi-protect

The RTPhelper is necessary as my SIP doorbell requires symmetric RTP.
As having two FFmpeg instances receiving and sending on the same UDP port does not work, I had to create an additional helper socket for this.
Without using symmetric RTP, my doorbell immediately stopped sending RTP audio when it started to receive RTP audio from some random port from FFmpeg.

… same port for TX and RX (symmetric RTP)

Sunoo · 2022-01-20T16:43:10Z

Oh, this is awesome. I’ll try to review this soon, but I expect I won’t merge it until I finish HKSV support (which shouldn’t be too long now).

nanosonde · 2022-01-20T16:46:14Z

Hi @Sunoo!

Of course, please take your time.
As this my first typescript/nodejs project, please forgive me.

BTW:
I have tested it here a couple of days and it works really stable.

nanosonde · 2022-01-20T16:50:05Z

If you want to try it out quickly without having a SIP doorbell, just just use this simple SIP client for Windows for example:
https://lite.phoner.de/index_en.htm

If you activate the debug view (you find it under "help"), you will see the SIP signalling.
Just skip the SIP account wizward.
The important thing in the config are the two IP addesses in the SIP config in TO and FROM.
BTW: in FROM you can also specifiy another port number after the IP address in case you have more SIP doorbells that need create a SIP stack each.

nanosonde · 2022-01-20T16:54:25Z

What is not implemented yet: the solution only does a direct SIP call to the doorbell. It does not issue a SIP REGISTER and it does not support authentication. This would be required to connect the SIP RINGING message to the doorbell characteristic.
However, as my doorbell also offers a HTTP webhook to be called on doorbell button press, this is enough for me.

Another interesting finding for my doorbell:
It is registered to my local Fritzbox (router with small SIP PBX inside). And the SIP registration to some other PBX works at the same time as a direct SIP call via IP. At least for my for my SIP doorbell.

mbrackjr · 2022-02-04T19:09:00Z

@nanosonde ; this sounds like a breakthrough! have been waiting for long to support my doorbird doorbell. I'm eager to test, but fail to understand how i could import this plugin while its waiting for pull-request, so i'll be patient until sunoo has the time to validate and accept.
For my understanding; is this function calling OUT to the doorbell or is it also able to have the doorbell directly do a SIP call inbound to homebridge on this plugin?

nanosonde · 2022-02-04T19:21:39Z

@nanosonde ; this sounds like a breakthrough! have been waiting for long to support my doorbird doorbell. I'm eager to test, but fail to understand how i could import this plugin while its waiting for pull-request, so i'll be patient until sunoo has the time to validate and accept.
For my understanding; is this function calling OUT to the doorbell or is it also able to have the doorbell directly do a SIP call inbound to homebridge on this plugin?

The current implementation calls OUT to the doorbell only. As a result the doorbell has to offer another way to signal the doorbell button press. For example my doorbell offers to also call a HTTP webhook. I use the HTTP webhook already provided by this plugin to send the iOS notification with snapshot.
If I then open the camera within homekit by clicking on the notification, it will open the Homekit camera as usual. While opening the camera, this SIP feature will then immediately call OUT and the doorbell automatically picks up the call immediately. Thus, two-way audio is also immediately available.
However, my doorbell only supports MJPEG video, I do not know what will happen with other codecs that might introduce non-synced audio video.

mbrackjr · 2022-02-04T19:27:52Z

Ok, sounds great; especially that you just click the default IOS notification makes for a seamless experience. Now I'm even more eager to test, hopefully Sunoo has time soon ;-)

ryan99alero · 2022-02-14T01:10:19Z

Took me a few minutes to figure out how to deploy into homebridge while using the config-UI-x solution. Just got my first successful call. Connection is to an Axis A8105-E door station. For those trying it uses UDP for protocol. Trying to figure out why the SIP payload is consuming so much bandwidth. Using more bandwidth on a peer to peer connection than my normal VOIP at office using asterisk and OPUS codec which is a wide band codec. Awesome work though. Curious if there will be plans to allow editing any core sip settings. Like a SIP.conf file for picking Sip Port, codec's, proxy, ICE, TURN, DTMF's, Media Encryption.

ryan99alero · 2022-02-14T01:45:30Z

Nanosonde, Curious on your tests. How much time was added to your connection time. When connecting I notice SIP connects pretty quick under 2 seconds but the Video part takes longer. Its taking 17 seconds to established full connection currently. When I was just using Sunoo's package without sip I had video connection down to about 8 seconds. May still be something on my end. Wondering if the feed is Pushing audio over both the H.264 feed and SIP and thats whats creating more bandwidth on my end.

ryan99alero · 2022-02-15T00:28:57Z

More playing around with your build and I can get a video without SIP streaming with RTSP audio in 2.5 seconds. Once I enable SIP its 13 seconds. I downloaded LinPhone app as is the only simple Peer-Peer SIP client I know of. Simple is the key word. That will connect to Axis Door Station SIP account and have full duplex audio in under 2 seconds. Guess I could try my current DoorBird DoorStation that the Axis will be replacing and see how it performs with SIP and Video.

longzheng · 2022-09-11T01:23:54Z

Thanks so much for your efforts. I'm just trying it out with my 2N Intercom which supports SIP

I set up my config as

            "cameras": [
                {
                    "name": "Intercom test",
                    ...
                    "sipConfig": {
                        "to": "sip:192.168.1.3", // ip of my intercom
                        "from": "sip:192.168.1.2" // ip of my homebridge server
                    }
                }
            ],

But I'm seeing a SIP INVITE error when I open the camera

[11/09/2022, 11:19:18 am] [Camera FFmpeg] sip INVITE request failed with status 481
[11/09/2022, 11:19:18 am] [Camera FFmpeg] [Intercom test] SIP INVITE failed: Error: sip INVITE request failed with status 481
[11/09/2022, 11:19:18 am] [Camera FFmpeg] [Intercom test] Starting video stream: 1280 x 720, 30 fps, 299 kbps (AAC-eld)
[11/09/2022, 11:19:18 am] [Camera FFmpeg] [Intercom test] FFmpeg exited with code: 1 and signal: null (Error)
[11/09/2022, 11:19:18 am] [Camera FFmpeg] [Intercom test] Stopped video stream.
[11/09/2022, 11:19:18 am] [Camera FFmpeg] sip BYE request failed with status 481

When I test a SIP direct call using MicroSIP to my intercom, it works and I see this in the MicroSIP logs

INVITE sip:192.168.1.3 SIP/2.0
Via: SIP/2.0/UDP 172.29.32.1:60420;rport;branch=z9hG4bKPjbf4cb12c764e4dc09fba4b4031e1f0cf
Max-Forwards: 70
From: <sip:172.29.32.1>;tag=ce897c027d3448a1b14b2b431649f2d8
To: <sip:192.168.1.3>
Contact: <sip:192.168.1.33:60420;ob>
Call-ID: 524ec458ee4c42fdb084033f2cfe7bdc
CSeq: 16740 INVITE
Allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS
Supported: replaces, 100rel, timer, norefersub
Session-Expires: 1800
Min-SE: 90
User-Agent: MicroSIP/3.21.2
Content-Type: application/sdp
Content-Length:   799

v=0
o=- 3871884119 3871884119 IN IP4 192.168.1.33
s=pjmedia
b=AS:2184
t=0 0
a=X-nat:0
m=audio 4000 RTP/AVP 9 8 0 101
c=IN IP4 192.168.1.33
b=TIAS:64000
a=rtcp:4001 IN IP4 192.168.1.33
a=sendrecv
a=rtpmap:9 G722/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-16
a=ssrc:1786122839 cname:5bf320fe605815c9
m=video 4002 RTP/AVP 99 98 100 101
c=IN IP4 192.168.1.33
b=TIAS:2000000
a=rtcp:4003 IN IP4 192.168.1.33
a=sendrecv
a=rtpmap:99 H264/90000
a=fmtp:99 profile-level-id=42e01e; packetization-mode=1
a=rtpmap:98 H263-1998/90000
a=fmtp:98 CIF=1;QCIF=1
a=rtpmap:100 VP8/90000
a=fmtp:100 max-fr=30; max-fs=580
a=rtpmap:101 VP9/90000
a=fmtp:101 max-fr=30; max-fs=580
a=ssrc:463995847 cname:5bf320fe605815c9
a=rtcp-fb:* nack pli

longzheng · 2022-09-11T05:06:34Z

src/sip-call.ts

+            if (response.status !== 408 || method !== 'BYE') {
+              this.log.error(
+                `sip ${method} request failed with status ` + response.status
+              )
+            }
+            reject(
+              new Error(
+                `sip ${method} request failed with status ` + response.status
+              )
+            )


While debugging my SIP error, I noticed there's an error reason that might be useful to log

Suggested change

if (response.status !== 408 || method !== 'BYE') {

this.log.error(

`sip ${method} request failed with status ` + response.status

)

}

reject(

new Error(

`sip ${method} request failed with status ` + response.status

)

)

if (response.status !== 408 || method !== 'BYE') {

this.log.error(

`sip ${method} request failed with status ${response.status} and reason ${response.reason}`

)

}

reject(

new Error(

`sip ${method} request failed with status ${response.status} and reason ${response.reason}`

)

)

After adding the reason to the log, I'm seeing the error as Call/Transaction Does Not Exist

[11/09/2022, 3:04:56 pm] [Camera FFmpeg] [Intercom test] SIP INVITE failed: Error: sip INVITE request failed with status 481 and reason Call/Transaction Does Not Exist

Have you tried running tcpdump on the OS you're running this solution on? I'd also suggest if your 2N intercom supports a packet capture to also capture there and download. If the 2N doesn't support it and its hard wired you could do port mirroring on your network switch and plug another computer into the switch on the port you do the mirror on and do either another tcpdump or just direct into wireshark. You'll want to put the nic on the computer you plugged into the "Mirrored Port" into promiscuous mode. Then open both wire captures into wire shark and look for sip registrations and check out the flow charts. Should give you another view into the issue.

Ah good idea, the 2N supports packet capture.

Here's a trace when this plugin makes a SIP call which fails.

INVITE sip:192.168.1.3 SIP/2.0 allow: INVITE, ACK, CANCEL, OPTIONS, BYE, REFER, NOTIFY, MESSAGE, SUBSCRIBE, INFO, UPDATE call-id: 278378 contact: <sip:192.168.1.2> content-length: 110 content-type: application/sdp cseq: 22 INVITE from: <sip:192.168.1.2>;tag=302689 max-forwards: 70 supported: replaces, outbound to: "SIP doorbell client" <sip:192.168.1.3>;tag via: SIP/2.0/UDP 192.168.1.2:5060;branch=z9hG4bK373682;rport v=0 o=192.168.1.2 3747 461 IN IP4 192.168.1.2 s=Talk c=IN IP4 192.168.1.2 t=0 0 m=audio 15086 RTP/AVP 0 INVITE SIP/2.0 call-id: 278378 content-length: 0 cseq: 22 INVITE from: <sip:192.168.1.2>;tag=302689 to: "SIP doorbell client" <sip:192.168.1.3>;tag user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.2:5060;branch=z9hG4bK373682;rport=5060 INVITE SIP/2.0 call-id: 278378 content-length: 0 cseq: 22 INVITE from: <sip:192.168.1.2>;tag=302689 to: "SIP doorbell client" <sip:192.168.1.3>;tag user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.2:5060;branch=z9hG4bK373682;rport=5060 ACK sip:192.168.1.3 SIP/2.0 call-id: 278378 content-length: 0 cseq: 22 ACK from: <sip:192.168.1.2>;tag=302689 max-forwards: 70 to: "SIP doorbell client" <sip:192.168.1.3>;tag via: SIP/2.0/UDP 192.168.1.2:5060;branch=z9hG4bK373682;rport BYE sip:192.168.1.3 SIP/2.0 call-id: 278378 content-length: 0 cseq: 23 BYE from: <sip:192.168.1.2>;tag=302689 max-forwards: 70 to: "SIP doorbell client" <sip:192.168.1.3>;tag via: SIP/2.0/UDP 192.168.1.2:5060;branch=z9hG4bK67662;rport BYE SIP/2.0 call-id: 278378 content-length: 0 cseq: 23 BYE from: <sip:192.168.1.2>;tag=302689 to: "SIP doorbell client" <sip:192.168.1.3>;tag user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.2:5060;branch=z9hG4bK67662;rport=5060

And here's a trace when MicroSIP succeeds

INVITE sip:192.168.1.3 SIP/2.0 allow: PRACK, INVITE, ACK, BYE, CANCEL, UPDATE, INFO, SUBSCRIBE, NOTIFY, REFER, MESSAGE, OPTIONS call-id: 4dd6e58708a84a8ebddc5b0349e428a0 contact: <sip:192.168.1.33:53499;ob> content-length: 364 content-type: application/sdp cseq: 24888 INVITE from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 max-forwards: 70 min-se: 90 session-expires: 1800 supported: replaces, 100rel, timer, norefersub to: <sip:192.168.1.3> user-agent: MicroSIP/3.21.2 via: SIP/2.0/UDP 192.168.1.33:53499;rport;branch=z9hG4bKPj2f7c6a5736cf442a87dc89a1a29f3b19 v=0 o=- 3871901177 3871901177 IN IP4 192.168.1.33 s=pjmedia b=AS:84 t=0 0 a=X-nat:0 m=audio 4000 RTP/AVP 9 8 0 101 c=IN IP4 192.168.1.33 b=TIAS:64000 a=rtcp:4001 IN IP4 192.168.1.33 a=sendrecv a=rtpmap:9 G722/8000 a=rtpmap:8 PCMA/8000 a=rtpmap:0 PCMU/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 a=ssrc:1122779591 cname:4d4a6ffd081c26cc INVITE SIP/2.0 call-id: 4dd6e58708a84a8ebddc5b0349e428a0 content-length: 0 cseq: 24888 INVITE from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 to: <sip:192.168.1.3> user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.33:53499;rport=53499;branch=z9hG4bKPj2f7c6a5736cf442a87dc89a1a29f3b19 INVITE SIP/2.0 call-id: 4dd6e58708a84a8ebddc5b0349e428a0 contact: <sip:192.168.1.3:5060> content-length: 0 cseq: 24888 INVITE from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 to: <sip:192.168.1.3>;tag=1119705459 user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.33:53499;rport=53499;branch=z9hG4bKPj2f7c6a5736cf442a87dc89a1a29f3b19 INVITE SIP/2.0 allow: REGISTER, INVITE, ACK, CANCEL, OPTIONS, BYE, INFO, NOTIFY call-id: 4dd6e58708a84a8ebddc5b0349e428a0 contact: <sip:192.168.1.3:5060> content-length: 232 content-type: application/sdp cseq: 24888 INVITE from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 to: <sip:192.168.1.3>;tag=1119705459 user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.33:53499;rport=53499;branch=z9hG4bKPj2f7c6a5736cf442a87dc89a1a29f3b19 v=0 o=- 1469393603 2233767474801975965 IN IP4 192.168.1.3 s=HIP 2.35.1.47.3 c=IN IP4 192.168.1.3 t=0 0 m=audio 9000 RTP/AVP 9 101 c=IN IP4 192.168.1.3 a=rtpmap:9 G722/8000 a=rtpmap:101 telephone-event/8000 a=fmtp:101 0-16 ACK sip:192.168.1.3:5060 SIP/2.0 call-id: 4dd6e58708a84a8ebddc5b0349e428a0 content-length: 0 cseq: 24888 ACK from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 max-forwards: 70 to: <sip:192.168.1.3>;tag=1119705459 via: SIP/2.0/UDP 192.168.1.33:53499;rport;branch=z9hG4bKPj8395320e8cda4a5ab740b331c87396db BYE sip:192.168.1.3:5060 SIP/2.0 call-id: 4dd6e58708a84a8ebddc5b0349e428a0 content-length: 0 cseq: 24889 BYE from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 max-forwards: 70 to: <sip:192.168.1.3>;tag=1119705459 user-agent: MicroSIP/3.21.2 via: SIP/2.0/UDP 192.168.1.33:53499;rport;branch=z9hG4bKPj0f16e470e636447e8c592da9c6c777fd BYE SIP/2.0 call-id: 4dd6e58708a84a8ebddc5b0349e428a0 content-length: 0 cseq: 24889 BYE from: <sip:192.168.1.33>;tag=33533c0e25d246fbb81c310c75d3f6f4 to: <sip:192.168.1.3>;tag=1119705459 user-agent: 2N IP Solo 2.35.1.47.3 via: SIP/2.0/UDP 192.168.1.33:53499;rport=53499;branch=z9hG4bKPj0f16e470e636447e8c592da9c6c777fd

Have you verified the port is open on the homebridge server running the plugin? I believe the SIP service is called baresip. Make sure in logs something else isn't attempting to attach to same port. I believe you've said you used the desktop sip app to test the camera but I didn't see anything about testing the plugin as well. Make sure your firewall is open. run this from a remote machine to make sure you can hit the box external. You can change the range from 1-65535 to be smaller just to cover your needed scope.

This command will show you what ports services are listening on.
sudo netstat -tulpn | grep LISTEN

This ran from remote machine toward the homebridge will let you know if you firewall or something else is stopping inward connection attempts.
nc -z -v 1-65535 2>&1 | grep -v 'Connection refused'

Turns out it wasn't an issue with the port.

Through adding a whole bunch of logging and trial and error, I narrowed it down to the to header having an empty tag in the INVITE request.

For example, this plugin was sending an INVITE request with the to header as

to: "SIP doorbell client" <sip:192.168.1.3>;tag

Whereas MicroSIP was sending

to: <sip:192.168.1.3>

If I modify https://github.com/Sunoo/homebridge-camera-ffmpeg/pull/1253/files#diff-9195b8330fc9c58adfffb0f687ffa35577548a16e2960e06b91f29c20db2be1eR181 to

params: this.toParams.tag ? this.toParams : undefined,

so it does not send an empty tag if it is undefined, then the INVITE chain all works correctly. I see a ringing and OK response afterwards.

Now I'm running into a separate issue where my FFmpeg is erroring starting up.

[12/09/2022, 8:52:44 pm] [Camera FFmpeg] [Intercom test] Starting video stream: 1280 x 720, 30 fps, 299 kbps (AAC-eld) [12/09/2022, 8:52:44 pm] [Camera FFmpeg] [Intercom test] FFmpeg exited with code: 1 and signal: null (Error)

I presume it's erroring trying to initialize the SIP call. I didn't have enough time today to figure out why but will aim to do more debugging soon.

Figured out the issue with the FFmpeg stream error which led to a bug https://github.com/Sunoo/homebridge-camera-ffmpeg/pull/1253/files#r969484626

However even after fixing this I'm still not getting the return audio working. Will need to debug further. It feels good chipping away at this every day after work.

I've been trying to debug why the "two-way" FFmpeg stream keeps crashing. I added debugReturn: true to my config and saw this in the logs

[14/09/2022, 8:44:48 pm] [Camera FFmpeg] [Intercom test] [Two-way] Stream command: C:\Users\Long\Desktop\homebridge-camera-ffmpeg-sipdoorbell\node_modules\ffmpeg-for-homebridge\ffmpeg.exe -hide_banner -protocol_whitelist pipe,udp,rtp,file,crypto -f sdp -c:a libfdk_aac -i pipe: -acodec pcm_mulaw -ac 1 -ar 8k -f rtp -payload_type 0 rtp://127.0.0.1:15128?rtcpport=15129 -loglevel level+verbose [14/09/2022, 8:44:48 pm] [Camera FFmpeg] [Intercom test] [Two-way] [sdp @ 000001bad9fe6980] [verbose] setting jitter buffer size to 500 [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [warning] Guessed Channel Layout for Input Stream #0.0 : mono [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Input #0, sdp, from 'pipe:': [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Metadata: [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] title : Talk [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Duration: N/A, bitrate: N/A [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Stream #0:0: Audio: aac, 16000 Hz, mono, s16 [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Stream mapping: [14/09/2022, 8:44:59 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Stream #0:0 -> #0:0 (aac (libfdk_aac) -> pcm_mulaw (native)) [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [error] pipe:: Unknown error [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [graph_0_in_0_0 @ 000001bada5aa3c0] [verbose] tb:1/16000 samplefmt:s16 samplerate:16000 chlayout:0x4 [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [format_out_0_0 @ 000001bada5ac800] [verbose] auto-inserting filter 'auto_resampler_0' between the filter 'Parsed_anull_0' and the filter 'format_out_0_0' [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [auto_resampler_0 @ 000001bada5ace00] [verbose] ch:1 chl:mono fmt:s16 r:16000Hz -> ch:1 chl:mono fmt:s16 r:8000Hz [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Output #0, rtp, to 'rtp://127.0.0.1:15128?rtcpport=15129': [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Metadata: [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] title : Talk [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] encoder : Lavf58.38.101 [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Stream #0:0: Audio: pcm_mulaw, 8000 Hz, mono, s16, 64 kb/s [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] Metadata: [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] encoder : Lavc58.70.100 pcm_mulaw [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] size= 0kB time=00:00:00.00 bitrate=N/A speed= 0x [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] No more output streams to write to, finishing. [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] size= 0kB time=00:00:00.00 bitrate=N/A speed= 0x [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [info] video:0kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] Input file #0 (pipe:): [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] Input stream #0:0 (audio): 0 packets read (0 bytes); 0 frames decoded (0 samples); [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] Total: 0 packets (0 bytes) demuxed [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] Output file #0 (rtp://127.0.0.1:15128?rtcpport=15129): [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] Output stream #0:0 (audio): 0 frames encoded (0 samples); 0 packets muxed (0 bytes); [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [verbose] Total: 0 packets (0 bytes) muxed [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [warning] Output file is empty, nothing was encoded (check -ss / -t / -frames parameters if used) [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [AVIOContext @ 000001bad9feaf40] [verbose] Statistics: 0 seeks, 0 writeouts [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] [AVIOContext @ 000001bad9fe7400] [verbose] Statistics: 356 bytes read, 0 seeks [14/09/2022, 8:45:10 pm] [Camera FFmpeg] [Intercom test] [Two-way] FFmpeg exited with code: 0 and signal: null (Error)

The notable error seem to be [Two-way] [error] pipe:: Unknown error which seems to imply something's wrong with the pipe input which I dumped as

v=0 o=- 0 0 IN IP4 192.168.1.141 s=Talk c=IN IP4 192.168.1.141 t=0 0 m=audio 14695 RTP/AVP 110 b=AS:24 a=rtpmap:110 MPEG4-GENERIC/16000/1 a=rtcp-mux a=fmtp:110 profile-level-id=1;mode=AAC-hbr;sizelength=13;indexlength=3;indexdeltalength=3; config=F8F0212C00BC00 a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:kPiXERZJq/CUIDqyz54SFAWKDav7MevvzenVBd4G

The IP address 192.168.1.141 is the phone that I'm testing with so that part all looks right. I'm not quite sure where/how the return audio stream is failing.

I've been debugging this more and am still stuck. I've tried changing to call a SIP client on my PC and running into the same issue with the two-way FFmpeg process erroring with Unknown error.

I've also tried changing from a pipe input to writing the SDP data into a file and loading that, same error, so I don't think it's an issue with the pipe but the stream directly.

From what I can understand, this seems to be the stream from the device loading the camera which in this case is my iPhone (the IP is also my iPhone).

@nanosonde not sure if you have any ideas about why the two-way input stream might be erroring like this?

Figured out the problem was with Windows Firewall. Because I was testing this plugin in a different directory using npm link, the ffmpeg.exe process wasn't allowed in the firewall to receive incoming connections which the RTP connection was. After I added it to the allow list the stream works.

Good catch. I was running on the assumption you were running Linux. To many unknowns / minimal support with Microsoft. You're braver than I am.

longzheng · 2022-09-13T11:10:03Z

src/streamingDelegate.ts

@@ -359,6 +434,15 @@ export class StreamingDelegate implements CameraStreamingDelegate {

      let ffmpegArgs = this.videoConfig.source!;

+      if (this.sipConfig) {
+        ffmpegArgs += // SIP audio via RTP
+        '-hide_banner' +


Suggested change

'-hide_banner' +

' -hide_banner' +

This was causing an issue with my FFmpeg stream since it was appending the -hide_bannerdirectly to the input URL without a space as a separate argument.

For example my config was

"source": "-i rtsp://192.168.1.3/h264_stream",

and the FFmpeg command was executed as

ffmpeg.exe -i rtsp://192.168.1.3/h264_stream-hide_banner -protocol_whitelist pipe,udp,rtp,file,crypto...

which leads to an error since rtsp://192.168.1.3/h264_stream-hide_banner is not the correct stream URL.

longzheng · 2022-09-20T10:14:49Z

src/sip-call.ts

+            to: {
+              name: '"SIP doorbell client"',
+              uri: this.sipOptions.to,
+              params: this.toParams,


Fixes issue with my 2N SIP doorbell not correctly responding to the INVITE if it receives an empty tag in the to: header.

Suggested change

params: this.toParams,

params: this.toParams.tag ? this.toParams : undefined,

For example

to: "SIP doorbell client" <sip:192.168.1.3>;tag

would not work correctly whereas

to: "SIP doorbell client" <sip:192.168.1.3>

works

KoljaV · 2022-10-24T07:26:23Z

Is there someone who could explain how I can install this plugin while using the config-ui-x solution ? Would love to test it…

mrMiimo · 2022-10-29T16:57:28Z

so if I understand correctly, is there support for SIP calls?
@nanosonde, please.. is there a way to test your fork?

mbrackjr · 2022-11-05T11:33:58Z

@nanosonde: Any idea how I can test this pull-request inside homebridge running on RPi while it's awaiting acceptance by @Sunoo

ryan99alero · 2022-11-07T19:15:15Z

@KoljaV In past I had to use CLI. Clone the repo using the brach command and recursive so you get dependencies if any. Then I'd run the NPM update, audit fix, outdated, install then do an NPM pack which converts the build you just created into a tgz file. Finally you'd use the NPM command line to install that package which will be seen in the Config UI admin portal. There may be a few more steps then what I've outlined below. This is just a few commands I saved as a reminder how to get started deploying from a plugin not listed in options or from a different branch.

git clone -b sipdoorbell --single-branch https://github.com/nanosonde/homebridge-camera-ffmpeg.git
npm-check-updates
npm update
npm audit fix
npm outdated -g --depth=1
npm install -g npm@latest
npm instal
npm audit fix
npm pack
sudo npm install -g homebridge-camera-ffmpeg-3.1.4.tgz

ryan99alero · 2022-11-07T19:17:49Z

@KoljaV in this thread there are several patches other's have done that aren't part of the sipdoorbell branch. You'd maybe want to fork this build on GitHub and apply whatever fixes you'd want that others have done.

Sunoo · 2022-11-07T19:19:31Z

If you guys can tell me for sure this works, I don’t have a strong problem with merging it somewhat blindly. That should make life easier for the folks who want to take advantage of it.

longzheng · 2022-11-07T21:49:23Z

@KoljaV In past I had to use CLI. Clone the repo using the brach command and recursive so you get dependencies if any. Then I'd run the NPM update, audit fix, outdated, install then do an NPM pack which converts the build you just created into a tgz file. Finally you'd use the NPM command line to install that package which will be seen in the Config UI admin portal. There may be a few more steps then what I've outlined below. This is just a few commands I saved as a reminder how to get started deploying from a plugin not listed in options or from a different branch.

I personally use npm link https://docs.npmjs.com/cli/v8/commands/npm-link/ which is easy to update/test since I just run a build and don't need to pack and install again.

longzheng · 2022-11-07T21:51:41Z

@KoljaV in this thread there are several patches other's have done that aren't part of the sipdoorbell branch. You'd maybe want to fork this build on GitHub and apply whatever fixes you'd want that others have done.

Yeah I was hoping the original submitter @nanosonde would accept the suggestions into his branch so it can be merged altogether. Otherwise maybe I can look at forking his PR, applying the changes and making a new PR as well.

If you guys can tell me for sure this works, I don’t have a strong problem with merging it somewhat blindly. That should make life easier for the folks who want to take advantage of it.

Based on my testing and suggestions, I think there are a few issues that might also affects others if they're not applied.

Sunoo · 2022-11-07T22:08:01Z

@longzheng If you’re able to correct them in a new PR, I’ll merge that.

ryan99alero · 2022-11-07T22:54:32Z

@longzheng I recently redeployed my homebridge VM. If you decide to put all into a PR. Let me know the branch and PR ID and i'll redeploy and see if it works for me. I have an AXIS A8105-E I just ordered to replace my DoorBird for better quality. I also have an Axis A8207-VE I got for Work but I'd imagine they both have the same SIP methodology. I'd planned on post upgrade to try and take the DoorBird apart and install a higher quality CMOS sensor. Otherwise I was just going to add your edits into a fork I made from @nanosonde branch.

longzheng · 2022-11-08T00:35:18Z

@longzheng If you’re able to correct them in a new PR, I’ll merge that.

@longzheng I recently redeployed my homebridge VM. If you decide to put all into a PR. Let me know the branch and PR ID and i'll redeploy and see if it works for me.

Yep I'll do that after work today.

longzheng · 2022-11-08T23:08:20Z

@Sunoo @ryan99alero I've opened the new PR #1355

Sunoo · 2022-11-08T23:09:18Z

Fantastic, closing in favor of #1355

nanosonde · 2022-11-14T08:13:35Z

Yeah I was hoping the original submitter @nanosonde would accept the suggestions into his branch so it can be merged altogether. Otherwise maybe I can look at forking his PR, applying the changes and making a new PR as well.

Thanks for working on this.
Personally I have no plans to work on this particular PR for this plugin anymore.
Instead I have been working on something new.

I made myself familiar with gstreamer and was able to get a complete homekit pipeline running with it instead of ffmpeg. Mainly because it supports RTP/SAVPF out of the box (RTCP PLI would generate a new key frame in the H264 encoder for example). Something not possible with ffmpeg. Also I got OPUS working after reading the Homekit spec. carefuly about the deviations from the RFCs. It is just in a PoC state in Python, but could probably directly be ported to JS as done here.

Then I came across this comment.
I have started to look into the code and observed that @koush has already solved so many things (RTCP feedback, OPUS, etc.) with scrypted and the system is much more flexible with respect to the plugin architecture. So I decided to drop all my work so far and I am currently analysing the options to get my SIP video doorbell working with scrypted.
Either by writing an own scrypted plugin or by providing some external solution which bridges between SIP and something that one of the scrypted plugins already support.

After koush's comment that I might have hit a feature gap (*1) in scrypted I am currently evaluating in creating a simple SIP<>ONVIF bridge based on my previous gstreamer experiences. Gstreamer already supports the ONVIF profile T backchannel stuff via RTSP and my Python experiments showed that it is really working.
However, as the scrypted ONVIF plugin requires a real ONVIF device to interrogate things like "GetProfiles()" etc. I am also searching for some simple ONVIF mock solution that makes the plugin believe it is talking to a real ONVIF device.
Another option that I have been experimenting with: provide some emulated unifi protect server based on FastAPI. Camera streaming would work with RTSP and the audio speaker channel would be realised via websocket.

(*1) The problem with all plugins is the fact that due to having video completely separated (RTSP/HTTP) from audio (SIP+RTP), it would be required to reconfigure the RTSP streaming once the audio connection to the camera is established too. However, to my knowledge reconfiguring RTSP streams during runtime is something that is specified in the RFC, but not very well supported.
A possible solution to this would be to use an always existing audio streams which just contains silence when the SIP audio is not connected. This could also be realized quite easy with the gstreamer audiomixer element.

Side note: if the video stream would have been also negotiated via SIP, it would have been easier to directly create a SIP plugin for scrypted based on older version of the ring plugin which used SIP to get video+audio streams from the cloud.

Too many possibilities....

mrMiimo · 2022-11-14T10:54:11Z

@nanosonde excellent reasoning ...
scrypted is a solid foundation on which to build anything.
If I understand your thinking, think of Sip to Sip calls (no proxy or registrar is involved), instead there are many intercoms that require access to a SIP server (https://www.linphone.org/sites/default/files/solutions-intercomsystems.pdf) ...

I don't know if it can help you, maybe even with video management, but lately I use Baresip (https://github.com/baresip/baresip) it's and very modular, it allows all types of calls (even SIP / SIP) manages a number infinite of audio and video codecs and can also be integrated with mqtt (useful for managing the call). I don't know if you know but baresip is like linphone in command line ...

I managed to integrate my intercom into homekit using alsa audio loopback, homebridge-ffmpeg and baresip ... audio quality is fantastic but video activation is very slow. In fact I only use the audio part and for the video part I add an encrypted rtsp video stream

nanosonde added 8 commits January 15, 2022 12:47

First working two-way audio SIP call

6b5c3f7

Cleanup

d3d5f2e

Cleanup and create SipCall instance at startup

5f4dddd

Added RtpHelper to fix problem of having two ffmpeg instances use the…

3b9ce58

… same port for TX and RX (symmetric RTP)

Add configuration via config file and cleanup

e6aebaf

Cleanup

bfa6026

Cleanup and add possibility to specify SIP port for SIP client

180bdcf

Cleanup

0a506ba

Cleanup

5536895

nanosonde mentioned this pull request Feb 11, 2022

Two-Way Audio seydx/homebridge-camera-ui#412

Open

kklem0 mentioned this pull request Apr 3, 2022

two-way audio for SIP based video doorbell ikalchev/HAP-python#407

Open

nanosonde mentioned this pull request May 9, 2022

Two way audio AlexxIT/WebRTC#243

Closed

nanosonde mentioned this pull request Aug 12, 2022

Homekit Secure Video #338

Open

longzheng reviewed Sep 11, 2022

View reviewed changes

longzheng reviewed Sep 13, 2022

View reviewed changes

longzheng reviewed Sep 20, 2022

View reviewed changes

longzheng mentioned this pull request Nov 8, 2022

Add support for two-way audio for SIP based video doorbell #1355

Open

Sunoo closed this Nov 8, 2022

nanosonde mentioned this pull request Nov 13, 2022

New plugin for SIP video doorbell - concept question koush/scrypted#427

Closed

	params: this.toParams,
	params: this.toParams.tag ? this.toParams : undefined,

Add support for two-way audio for SIP based video doorbell #1253

Add support for two-way audio for SIP based video doorbell #1253

Conversation

nanosonde commented Jan 20, 2022 • edited Loading

Sunoo commented Jan 20, 2022

nanosonde commented Jan 20, 2022 • edited Loading

nanosonde commented Jan 20, 2022

nanosonde commented Jan 20, 2022 • edited Loading

mbrackjr commented Feb 4, 2022 • edited Loading

nanosonde commented Feb 4, 2022 • edited Loading

mbrackjr commented Feb 4, 2022

ryan99alero commented Feb 14, 2022

ryan99alero commented Feb 14, 2022

ryan99alero commented Feb 15, 2022

longzheng commented Sep 11, 2022 • edited Loading

longzheng Sep 11, 2022 • edited Loading

Choose a reason for hiding this comment

ryan99alero Sep 11, 2022

Choose a reason for hiding this comment

longzheng Sep 11, 2022 • edited Loading

Choose a reason for hiding this comment

ryan99alero Sep 11, 2022

Choose a reason for hiding this comment

longzheng Sep 12, 2022 • edited Loading

Choose a reason for hiding this comment

longzheng Sep 13, 2022

Choose a reason for hiding this comment

longzheng Sep 14, 2022 • edited Loading

Choose a reason for hiding this comment

longzheng Sep 18, 2022 • edited Loading

Choose a reason for hiding this comment

longzheng Sep 23, 2022

Choose a reason for hiding this comment

ryan99alero Sep 23, 2022

Choose a reason for hiding this comment

longzheng Sep 13, 2022

Choose a reason for hiding this comment

longzheng Sep 20, 2022 • edited Loading

Choose a reason for hiding this comment

KoljaV commented Oct 24, 2022

mrMiimo commented Oct 29, 2022 • edited Loading

mbrackjr commented Nov 5, 2022

ryan99alero commented Nov 7, 2022

ryan99alero commented Nov 7, 2022

Sunoo commented Nov 7, 2022

longzheng commented Nov 7, 2022

longzheng commented Nov 7, 2022

Sunoo commented Nov 7, 2022

ryan99alero commented Nov 7, 2022

longzheng commented Nov 8, 2022

longzheng commented Nov 8, 2022

Sunoo commented Nov 8, 2022

nanosonde commented Nov 14, 2022 • edited Loading

mrMiimo commented Nov 14, 2022 • edited Loading

nanosonde commented Jan 20, 2022 •

edited

Loading

nanosonde commented Jan 20, 2022 •

edited

Loading

nanosonde commented Jan 20, 2022 •

edited

Loading

mbrackjr commented Feb 4, 2022 •

edited

Loading

nanosonde commented Feb 4, 2022 •

edited

Loading

longzheng commented Sep 11, 2022 •

edited

Loading

longzheng Sep 11, 2022 •

edited

Loading

longzheng Sep 11, 2022 •

edited

Loading

longzheng Sep 12, 2022 •

edited

Loading

longzheng Sep 14, 2022 •

edited

Loading

longzheng Sep 18, 2022 •

edited

Loading

longzheng Sep 20, 2022 •

edited

Loading

mrMiimo commented Oct 29, 2022 •

edited

Loading

nanosonde commented Nov 14, 2022 •

edited

Loading

mrMiimo commented Nov 14, 2022 •

edited

Loading