Question on using just the jvb without the rest of jitsi-meet

[Luc35b]

Is it possible/feasible to setup and use only the jvb sfu server without the rest of jisti meet using plain webrtc and our own meeting constructs/UI instead of lib-jitsi-meet and then something like socket.io for signaling instead of prosody?

[saghul]

Technically possible, but the REST API for the bridge is undocumented and there is no helper library :-(

[Luc35b]

@saghul I see so would you have any suggestions or recommendations on what to use to as an sfu to maintain performance as participant counts scales for a meeting application instead of jvb while still having the ability to customize the ui and webrtc constructs?

[saghul]

I'm heavily biased here of course :-)

If you need your own UI lib-jitsi-meet is a great option, as is has all the signaling and scaling figured out.

Don't be startled by XMPP / prosody, it's just an implementation detail, and if you roll your own everything on top of socket.io you'd end up reinventing most of the same constructs that are necessary.

Good luck with your project!

[Luc35b]

@saghul Ah ok its just using lib-jitsi-meet I've run into problem like low audio/video quality and a slight delay with uncertainty when from a p2p to sever connection so I was trying to find other solutions. Also in trying to add a remote control feature there's also been issues with latency using the sendMessage data channel unless you know a way to add another custom data channel for just remote control or another solution for this.

[saghul]

just using lib-jitsi-meet I've run into problem like ;ow audio/video quality and a slight delay

Hard to tell what could have gone wrong without more details about your setup and configuration. LJM is (reasonably) low level, so a number of knobs need to be turned for optimal performance, depending on your use case.

un certainty when from a p2p to sever connection

Generally speaking you shouldn't need to worry about that. P2P is only used for 1-to-1 calls, as an optimization. The moment a 3rd participant joins the call it will seamlessly transition to the JVB.

Also in trying to add a remote control feature there's also been issues with latency using the sendMessage data channel

sendMessage uses the XMPP signalling for transport, which is not well suited for high volume transfers, you can use sendEndpointMessage instead, which uses the JVB datachannel as a transport.

You are of course welcome to use whatever software better suits your needs such as MediaSoup or Janus, both made by dear friends of mine; but I'd be remiss if I didn't point out the above ;-)

[Luc35b]

Thanks you so much for the suggestions, I'll investigate/experiment more and let you know if I have a follow up question or solve the issues!

[Luc35b]

@saghul Since sendEndpointMessage is deprecated, checking the "sendThroughVideoBridge" condition instead fixed the latency issue. I'm still looking into video/audio optimization but was wondering if I was able to create new data channels on the lib-jitsi-meet constructs to separate out chat and remote control when sending data with jvb. So far I've tried the approach of trying to access the RTCPeerConnection object through the JitsiConference object with "room?.current?.getActiveMediaSession()?.peerconnection?.peerconnection" then calling createDataChannel on this but nothing seems to be passing through the server with this. Is there any specific way to get this to work?

[saghul]

Since sendEndpointMessage is deprecated, checking the "sendThroughVideoBridge" condition instead fixed the latency issue.

I forgot we deprecated that API. Under the hood they do the same thing though. Glad to hear this helped with the latency issue, that's what I was expecting.

was wondering if I was able to create new data channels on the lib-jitsi-meet constructs to separate out chat and remote control when sending data with jvb.

This is currently not possible. That said, they are already separate. The chat goes voer XMPP and endpoint messages over the JVB and are free-form.

So far I've tried the approach of trying to access the RTCPeerConnection object through the JitsiConference object with "room?.current?.getActiveMediaSession()?.peerconnection?.peerconnection" then calling createDataChannel on this but nothing seems to be passing through the server with this. Is there any specific way to get this to work?

That won't work.

[Luc35b]

Ah ok I was starting to suspect it wouldn't. I was hoping to have multiple data channels so a file transfer feature could also be separated to its own data channel so hopefully multiplexing the one channel wont cause any significant performance issues.

[saghul]

I don't think it would make a huge difference, your bandwidth still needs to be split between the 2.

[4n1m4t10n]

Hello there! I am Jose, part of Lucas' team for this project. I was wondering what kind of options LJM has specifically that will help us deal with low quality/laggy video feeds. Thank you for the heads up!

[saghul]

Hey @4n1m4t10n ! There are a number of things that can affect quality:

• What receiver constraints are you setting?
• What constraints are you creating local tracks with?

Those 2 are a good start.

[shwetd19]

Hi @saghul, quick question: for low-quality video and lag, what specific settings in lib-jitsi-meet would you recommend tweaking first? Like, are there key configs for bandwidth, resolution, or codecs that usually make the biggest difference?

Thanks!

[saghul]

The answer to the question I posted in #2627 (comment) would help.

If you request 180p for every participant, well, that's what you'll get, for example. Likewise, if you request 4k for every participant in a meeting, things are not going to go alright...

[shwetd19]

Hey @saghul , For optimal performance, I suggest setting a dynamic resolution constraint starting at 720p for 1-2 participants and gradually scaling down to 360p or 180p as more participants join, which we can implement using conference.setReceiverVideoConstraint(height) based on participant count.

[saghul]

That is what Jitsi Meet does, roughly. This is a library so we cannot assume what use case the user wants.

[Luc35b]

@saghul We tried setting some config options to improve our video/audio quality, specifically while screen sharing, setting what's bellow, but we find quality to be inconsistent and to degrade in quality as remote tracks for other users. Do you happen to know what could be going wrong or if we are missing a crucial config? :

const confOptions = {
desktopSharingFrameRate: { min: 30, max: 60 },
p2p: { enabled: true }, testing: { capScreenshareBitrate: 0 },
constraints: {
video: { height: { ideal: 1080, max: 1080, min: 720 },
width: { ideal: 1920, max: 1920, min: 1280 },
frameRate: { min: 30, ideal: 60, max: 60 } },
audio: { echoCancellation: true, noiseSuppression: true },
channelLastN: -1, resolution: 1080, codecPreferenceOrder: ['AV1', 'VP9', 'VP8', 'H264'] } };

room.current.setDesktopSharingFrameRate(30);
room.current.setReceiverVideoConstraint(720);

When creating local tracks for screen sharing:
JitsiMeetJS.createLocalTracks({
devices: ['desktop', 'audio'],
constraints: { video: { mandatory: { minFrameRate: 30, // Force minimum FPS
maxFrameRate: 60, // Try forcing 60 FPS
width: 1920, height: 1080 } } } })

[damencho]

What content are you screen sharing? If it is text you should lower the framerate to 5, this way the content will have a setting detail and text will be in higher quality. Screensharing with higher framerate is for video content which lacks that detail flag.

[Luc35b]

@damencho Since we've added a remote control, we wanted the video quality and latency to be as optimal as possible but are struggling to achieve this balance, with consistent quality being a particular issue. It would be ideal to not have to lower the fps to 5 so things like visually tracking the mouse have a better feel.

[jallamsetty1]

Hi @Luc35b I suggest, removing the desktopSharingFrameRate constraint from your config.js.
In the absence of this constraint, the client runs in the auto mode and the fps will vary based on fps of the content that is being shared. If you use the SVC codecs, i.e., AV1 and VP9, you will get the highest resolution stream with temporal scalability when needed.
By calling room.current.setDesktopSharingFrameRate(30); you are requesting the client to send all 3 spatial layers so the bridge might end up sending lower resolutions based on their available downlink.
Also please make sure that you are sending proper receiver constraints from the receiving endpoints to the bridge. Screenshare sources should be added to onStageSources for the bridge to prioritize SS over other sources when it is doing bandwidth allocation.

[4n1m4t10n]

Hello there. I am also working on this project alongside Luc, and wanted to clarify some steps we are taking to fix this issue. The following function is called when a screen share track is created:

const setReceiverConstraints = (desktopTrackId: string) => {
const constraints = {
lastN: 20,
selectedSources: [],
onStageSources: [desktopTrackId],
defaultConstraints: { maxHeight: 720 },
constraints: {
[desktopTrackId]: { maxHeight: 720 }
}
};
room.current?.setReceiverConstraints(constraints);
};

Unfortunately, removing the room.current.setDesktopSharingFrameRate(30); when a room is initialized seems to cap the screen share video at 5 FPS from other participants' perspective. We are not sure what other constraints are causing this

[Luc35b]

Hello there. I am also working on this project alongside Luc, and wanted to clarify some steps we are taking to fix this issue. The following function is called when a screen share track is created:

const setReceiverConstraints = (desktopTrackId: string) => { const constraints = { lastN: 20, selectedSources: [], onStageSources: [desktopTrackId], defaultConstraints: { maxHeight: 720 }, constraints: { [desktopTrackId]: { maxHeight: 720 } } }; room.current?.setReceiverConstraints(constraints); };

Unfortunately, removing the room.current.setDesktopSharingFrameRate(30); when a room is initialized seems to cap the screen share video at 5 FPS from other participants' perspective. We are not sure what other constraints are causing this

@jallamsetty1 to add some more context to @4n1m4t10n comment, we also tried adding the 'desktopTrackId' to selectedSources and using the screen share track's '_sourceName' using '.getSourceName()' instead of the track id. Were aren't quite sure what we are doing wrong as while the quality with 2 participant via p2p is good, once a 3rd participant joins and the jvb is used we experience the halving in video quality referenced here: #1703

[4n1m4t10n]

One last thing to add is that we are fairly certain this is a Jitsi issue and not a network one, since our other audio and video tracks seem good from other participant's perspectives contrary to screen share. Thank you so much for your time!

[jallamsetty1]

@4n1m4t10n @Luc35b Yes desktopTrack's sourceName should be added to only onStageSources. Can you please share the receiver constraints sent out on the bridge channel by the receiver and the also the sender constraints received by the sender? You can simply share the sender and receiver's browser console logs for me to check.

[4n1m4t10n]

@jallamsetty1 Here are the sender and receiver video constraints with two people only from both the person screen sharing and the person receiving it:

Here are the constraints after a third participant shows up:

Here is the screen share's resolution getting halved after a third person joins:

[jallamsetty1]

As per the log, the client is still configured to prioritize frame rate which is a consequence of room.current.setDesktopSharingFrameRate(30)
Also, the source names you are passing to setReceiverConstraints look wrong to me. They should be of the format
<endpointID>-v<trackIndex> as shown below.

[Luc35b]

@jallamsetty1 We fixed the source names being passing to setReceiverConstraints as we accidentally left it as trackId on when setting this for remote tracks. We still found getting rid of 'room.current.setDesktopSharingFrameRate(30)' causes the frame rate of the screen share to stay at <5 but the quality halving issue goes away. We also tried cranking the constraints to asking for 2k quality while leaving 'room.current.setDesktopSharingFrameRate(30)' which allowed us to maintain both quality and frame rate. Is their a way for us to get this balance though without requesting such a high resolution/pixels or using 'room.current.setDesktopSharingFrameRate(30)'?

[jallamsetty1]

@Luc35b Are you passing desktopSharingFrameRate min/max in the config you are passing JitsiMeetJS.init()?
Cranking up the receiver constraints to request higher resolutions does nothing for SS, the client always encodes the full resolution of the captured tab/window/desktop but the general practice is to request -1 (for unlimited).

If you go with the defaults and not overwrite the settings, the client should run in auto mode, i.e., the captured fps will depend on the fluidity of the content being shared. If you are sharing a slideshow, the fps will mostly stay low. If you are sharing a video, then you should see the send fps adapt automatically.

[Luc35b]

We tried removing room.current.setDesktopSharingFrameRate(30); and desktopSharingFrameRate min/max from our configs but the fps doesn't seem to adapt even when a video is being screen shared. By adding room.current.setDesktopSharingFrameRate(30); back in and increasing the maxHeight though, we are getting a more consistent screen share video quality and frame rate we are currently satisfied with.

[saghul]

Don't we set maxHeight to 9999 to sidestep some Chrome bug?

[jallamsetty1]

Don't we set maxHeight to 9999 to sidestep some Chrome bug?

Yes we set to 99999 only when the desktopSharingFrameRate.max <= 5 and that improves the capture resolution. That is on the capture side though. @Luc35b is talking about the receiver constraints for SS sources.

[jallamsetty1]

@Luc35b it could be some other issue with your client. On meet.jit.si, I get up to 25 fps when I share a youtube video and we use the auto mode there - https://meet.jit.si/config.js.