Relax 1000-user limit on text chats

[bcpierce00]

Just making sure to confirm that / when this is fixed.

[jon-bell]

This is definitely, definitely not fixed

[bcpierce00]

High priority for SPLASH (and CSCW?), then!

[jon-bell]

From Twilio:

The procedure is to reach out here and I will work with you. You can reply to this message chain even after the Support ticket has been closed. A new ticket will be opened on you behalf that links to this original ticket.

There is however a maximum limit set at 3,000 at this time. Keep in mind that increasing the limit to this volume also incurs a tradeoff between volume and performance.

Let's just hope we don't cross 3k :)

[bcpierce00]

Good. But...

How confident are we that 3k is enough? Given that SPLASH and ECOOP are colocated, I could actually well imagine more than that number of participants...

[jon-bell]

I think that if 3k is a problem, we need to move out of Twilio and into something else. We can also make it somewhat better by removing users from channels when they are offline, so that our limit is 3k concurrent users (rather than 3k total users, including users who are subscribed to a channel but offline).

[bcpierce00]

We can also make it somewhat better by removing users from channels when they are offline, so that our limit is 3k concurrent users (rather than 3k total users, including users who are subscribed to a channel but offline).

This might make it enormously better!

[jon-bell]

And, it might not work so well, since joining/leaving a channel is an expensive operation - each channel has a limit of 30 actions-per-second (where an action is a subscribe/unsubscribe or message send/update/delete). This would be a bottleneck if 30 people all open Clowdr at the same time (and each of them try to join a whole bunch of channels).

Also, we would need to manage read/unread counts on our own (not in Twilio).

[bcpierce00]

Too bad. This is sounding like a problem...

[EdNutting]

I suspect for our purposes, operating our backend as a read-only mirror of large Twilio chats will be both easy and more than adequate. It would operate something like this:

1. Our node backend turns on the onMessageSend Webhook
   • This will give us a notification for every message sent in every channel regardless of size.
   • The alternative but even more costly approach would be for the node backend to subscribe to channels as users - not recommended for various good reasons.
2. We define a list of "large channels" via Instance Configuration. (Pre-load this into memory in node).
3. When an onMessageSend webhook arrives, we check its target channel. If it is in the list of "large channels", we store a copy of the message in our database.
4. When a user opens a chat client side, we again check it against the list of large chats.
   • If it isn't in the list, we operate as we currently do.
   • Else, it is in the list, so:
     • We don't subscribe the user to the channel.
     • Instead, we fetch the messages from the database (via the new client-side caching system), and display them to the user.
     • Via the client-side caching mechanism, the chat component will subscribe to Live Query, enabling them to receive new chats when they arrive.
     • If the user tries to send a message, then we create a Twilio client and subscribe them to the channel, then we send the message. (This is the same as currently, except the creation/subscription of the client has been deferred).
     • After X minutes of not sending a message, the user is automatically unsubscribed from the channel.
     • Even when subscribed to a large channel via Twilio, the user continues to receive messages via Live Query (this will present more consistent behaviour - effectively hiding when they're subscribed/unsubscribed).

It would be worth checking the Parse DB load of doing this after my new client-side caching system is ready (since the client side cache could cache such read-only channels thus eliminating duplicate downloads of lots of messages). Also, only fetching recent messages unless someone scrolls back in time would reduce load for our systems.

[EdNutting]

For large channels, read/unread counts could be easily emulated - for each user for each large channel, store in Parse DB the last time the user opened that channel, then client side count the number of messages since that time. This would utilise a read-modify-write db transaction.

[EdNutting]

I'm not ignoring Rocket.Chat by the way - I've been digging through their scaling documentation, etc. We would either use Rocket as SaaS or as a custom install. The latter is beyond our ability - we would need a full time devops person to even get that running properly. The former is feasible, but: using Rocket.Chat removes our ability to customise the chat. As far as I can see, to integrate it with the "conference program"-system would be tricky. I don't think it will yield the kind of smooth UI we are aiming for.

[jon-bell]

@EdNutting I like your design. I think that we might want to make some notification layer on top of Twilio anyway eventually to help manage @ mentions, and eventually deliver daily or hourly digests by email.

[EdNutting]

Yes, I think notifications would be a good feature (and beyond Twilio's chat API as there are things like "watched sessions" that we might want to notify about.)

Refs:

• Google (based on Firebase): https://codelabs.developers.google.com/codelabs/pwa-integrating-push/index.html?index=..%2F..dev-pwa-training#1
• Mozilla: https://serviceworke.rs/push-get-payload.html