New chat-based communication channel to replace Gitter/Matrix

[andrii-i]

This issue is created on behalf of Jupyter Media Strategy working group (charter). The goal of this issue is to get feedback from the community on replacing Gitter as a primary / official chat-based communication channel of Project Jupyter. Please share your thoughts and opinions.

Please note that change is intended for the whole Project Jupyter, not for a certain subproject. Issue about this at jupyter/governance:

• New chat-based communication channel to replace Gitter/Matrix for Project Jupyter jupyter/governance#182

Problem

Chat-based communication channels offer possibility of real-time interaction and rapid feedback which provides additional utility and value to open source projects compared to forums like Discourse. Gitter historically served as a main/most popular chat-based communication channel for the Project Jupyter community. Based on discussions during weekly Project Jupyter calls, after Gitter migrated to Matrix it's usability and popularity is not at a satisfactory level anymore (see #213 (comment) for for more details).

Proposed Solution

Create new chat-based communication channel as a replacement for Gitter.

Potential options:

• Discord
  • Pros: household name, familiar to majority of users
  • Cons: closed source
• Zulip
  • Pros: open source, can be self-hosted. Potentially better threading
  • Cons: less popular
• Zulip for technical communication, Discord for community communication (as mentioned here)

I prefer Discord due to popularity and ease of adoption.

[andrii-i]

Slack is another popular chat-based communication app but it has a rather limited free tier that only saves 90 days of message history. In paid tier organization pays a fixed sum for each user which is not sustainable for Project Jupyter.

https://app.slack.com/plans/T04F8K3FZB5

[astrojuanlu]

FYI https://www.linen.dev/ turns Slack and Discord into public ans searchable archives, we use it on Kedro and works well

[jtpio]

Thanks @andrii-i for starting this discussion 👍

Discord is one of the best options at the moment. Many open source communities have been using it for a long time already like Rust, Yarn, Fedora and many more: https://discord.com/open-source.
It would also be possible for the Jupyter Discord to be verified:

It would also be more productive to go where people are (on Discord), than trying to setup something different that would require a different account or workflow and would most of all add friction.

Sure Discord is closed source but this is likely not a blocker. We already use GitHub (which is also closed source) for the Jupyter projects because this is where people are.

[thibaultamartin]

Hi, I hope it's not inappropriate for me as an Element employee to slide in this discussion. I'm obviously biased, but my goal is not to convince you to stick on Matrix no matter what :)

I couldn't find a list of what's been problematic with Matrix. It would be tremendously helpful to know what is biting the community and see what solutions we have, or maybe shed some light on what we are already doing to fix it and when to expect improvement.

Additionally, and because Matrix is all about interoperability, the last thing we want is to lock communities in. t2bot.io is a service supported by the Matrix.org Foundation that hosts several bridges. They offer a free bridge between Matrix and Discord. This can be a good solution to:

• avoid a "hard move" of the community. People can move at their own pace
• allow folks to use their platform of choice
• keep a foot in the OSS world in case Discord changes their policies in any way that is not satisfactory

[krassowski]

I couldn't find a list of what's been problematic with Matrix. It would be tremendously helpful to know what is biting the community and see what solutions we have, or maybe shed some light on what we are already doing to fix it and when to expect improvement.

I do not know what do others experience, but for me the default Element experience has severely degraded since the migration from the old gitter UI:

• things like "Gitter is open in another window. Click "Continue" to use Gitter here and disconnect the other window.", click on it
• each time I open the page I get annoying "Set up Secure Backup" with no obvious way to discard it (we were using gitter as a transient chat; I do not care about backups there, and if there are backups for repo channel these should not bother every user who)
• The notifications for unread topics do not ever go away and require clicking on room three dots "Mark as read" (I only now found this options, why is it not exposed next to notifications and threads icons?); I do not want to scroll through hundreds of messages to find the one that I did not read
• Loading is slow; going back in history is often seeing the spinner (which makes the point above especially annoying)
• I think I no longer receive email notifications on mentions; I recently tried changing settings but no one mentioned me since so maybe it is just a problem that the default settings were not migrated and users now needed to opt-in
• There is a lot of noise I do not care about: X joined the room, Y changed their profile picture, someone changed their name, icons of 20 people who read a message "Seen by" (there is a benefit trade-off here, and it drops as the number of users in channel increases)
  
• Rooms:
  • Lots of puzzling "Empty room" rooms; chats being duplicated
  • Rooms being at the bottom, People being at the top (for the project chat switching between jupyter/jupyterlab/jupyterhub channels this is another inconvenience); the icon selector does not help as our icons are all similar
  • Rooms and people being mixed up
    
• Notifications in the panel not rendering names but user IDs with random strings which makes it hard to understand who is speaking:
  
• Distractions: features we don't use but are showing up as icons (calls, video - these are disabled in the project rooms so why show them?) - these are not shown in guest view via gitter

[thibaultamartin]

Thanks for the heads up @krassowski, it's important for us to know what issues people are running into. Those are recurring issues that we're aware of and actively trying to fix. I don't want to derail the issue, but I think it's helpful for your decision process to know what you can expect.

Known problems and fixes on the horizon

I'd say there are three major areas where you're being bitten:

• A confusing UI
  • The backup thing you mention is not about backing up the channels but about encryption keys backup, which is irrelevant in your case of public rooms that are not encrypted
  • Threads can be counter intuitive to use
  • The notifications panel is broken. We're investigating various ways to update this panel not only to make it work, but also to make it easier to use. We want to make it a kind of inbox where you can catch up with notifications when you've been away.
• Noise issues
  • The notification default settings are very noisy by default, which can be draining. You can fix this in the client settings already (Settings > Notifications), though I'll grant you this needs to be fixed for everyone by default. We're actively working on it and you can have a look at the new much simpler notification settings we're experimenting with on develop.element.io > Settings > Labs
  • The timeline default settings can be noisy for some. This can be changed in Element settings under Settings > Preferences > Timeline section. I don't think there's a one size fits all here.
• Protocol issues
  • There is no Canonical DM in Matrix. Historically it was possible to create duplicate DMs with a same person. Element is safeguarding it much more now, preventing it from creating a duplicate room if you want to start a new DM with someone you already talk to. Unfortunately, this doesn't fix the past rooms.
  • The sync is slow, which can be frustrating. Fortunately this is fixes in sliding-sync, a new sync method that is nearly instantaneous. It's already in production on some instances, and our new client (Element X) can use it.

Element has dedicated a lot of its resources on creating brand new clients to make the overall experience much better. Those are the Element X (mobile) clients that we're going to advertise more widely in the coming days.

The price for it has been a bit less love for Element Web and Desktop. As a result the issues you mentioned (and that are well known) have been making slower progress than we'd want to. With the Element X clients out we will give Element Web more love, to fix those issues.

There's nothing unfixable in these issues and we're working on it. It won't happen overnight though, so if they have become entirely unbearable, I'd suggest to keep a bridge to Matrix (via the free t2bot.io bridge) so your community doesn't go through a hard move, people who want to rely on OSS tools can do it, and you keep your project conversation history on OSS tools that are not subject to unilateral policy changes.

[isabela-pf]

If we plan to migrate, my personal, informal vote would be in favor of Zulip. While it historically isn't a strict requirement, I prefer that it is open source. I also think its thread-style is much more compatible with the quantity and variety of communications Jupyter has had on Gitter in my experience; though I will note it is comparable to the experience of the Google Groups/Jupyter Mailing List.

Is there a reason we'd prefer to start a new place to communicate over using a similar one we already have—like the Jupyter Mailing list—that I'm unaware of?

[krassowski]

Many open source communities have been using it for a long time already like Rust, Yarn, Fedora and many more: https://discord.com/open-source.

Rust also uses Zulip: https://rust-lang.zulipchat.com/. It would be valuable to learn from them how these two stack against each other in practice. From their docs it seems they use Discord for community and less technical teams, while the core dev teams and working groups use Zulip.

Zulip also offers free plan for open source: https://zulip.com/for/open-source/. I would be inclined to run a trial with it and see how it works for us. I use Discord and it seems fine for community building but is a bit heavier to run (not web-first in a sense); I do not want to have it on at all times.

[krassowski]

Mailing list does not offer the same kind of communication as gitter used to - quick online chat which may be answered instantaneously. I would not send many of the quick coordination messages that I do send on gitter if I had to send them via the mailing list. Both Zulip and Discord fulfil this role well. Also, mailing lists have distinct spam issues which lead to some messages being wrongly flagged (and even now some of my messages get blocked on our mailing lists before being manually reviewed, so I personally would be very unhappy if we were to restrict ourselves to that).

[krassowski]

@thibaultamartin thanks for the extra context. It is very valuable to have you in this discussion.

[yuvipanda]

What would the scope of this be for? Would this include the JupyterHub / Binder communities as well, or just JupyterLab?

[andrii-i]

@yuvipanda Change is intended for the whole Project Jupyter, not for a certain subproject. I posted here in JupyterLab team compass because topic of new chat-based communication channel to replace Gitter/Matrix was raised during JupyterLab calls a number of times and received a lot of interest. Editing issue description to clarify scope, also created issue at the appropriate scope jupyter/governance#182.

[krassowski]

We did discuss a possibility of trying out different platforms within JupyterLab first before attempting to suggest a move on the Project-wide level, just to see if the alternatives work out to be any better in the first place. But of course, it is better to discuss in Project-wide repo from the beginning to invite more comments from across subprojects. Though to be honest I do not know if governance repo is the most appropriate, I am worried that some contributors may not feel like sharing their opinion there because they are not part of the governance?

[ivanov]

Thank you for making an issue to discuss this.

Before gitter, we used (and paid for) hipchat for about 18 months (Feb 2013 - October 2014). This was both to facilitate work during the first notebook grant (public internal communication) and to do live troubleshooting and impromptu question & answer sessions for developers. I think as time went on, live troubleshooting and Q & A has become the primary use case for gitter.

Mailing lists, [IPython-User] and [IPython-dev] were historically the primary communication mechanisms among the user community and for technical discussions, respectively, and they were consolidated to one list in 2013. The unified -dev mailing list receives little traffic, these days, and my impression is that the work happens in PRs and discussed via issues here on github (in various repos).

Whatever the issues currently plaguing the gitter/matrix changes, it's only a matter of time before discord rocks the boat in some unfortunate direction (as slack is brilliantly demonstrating recently), and zulip being open source might be better, though honestly it is unlikely we would run our own instance, so we would just be subject to whatever choices they make for their hosted offering . I am of the opinion that the only way to win this game is not to play :) , but I wanted to encourage us to express and agree on what problem we're trying to solve with a chat based communication channel.

[yuvipanda]

Given that this is meant to be for the whole project, I'd love for someone with rights to move all the comments already here to jupyter/governance#182 and the discussion can continue there :)

[blink1073]

@meeseeksdev migrate to jupyter/governance

[blink1073]

Let's just close this one in favor of jupyter/governance#182, which already references this issue.

[andrii-i]

Thank you @blink1073