Initial support for Raise Hand feature

[mgcm]

Initial draft to implement #130

This implementation initially used LiveKit's API that allows for participants to publish data.

It is now using reactions to call membership events and redactions of said event to toggle raising of hands.

Planned features:

• show order of hand raises (1st, 2nd, etc)
• add a timer that shows how long the hand is raised

ec-raise-hand.mp4

[fkwp]

Nice :-)

This implementation uses LiveKit's API that allows for participants to publish data. I'm interested in discussing if it would make more sense to use extensible matrix events instead.

MatrixRTC is designed to not rely on the actual RTC backend (today it's LiveKit; tomorrow the-next-best-thing) it makes sense to implement those kinds of features with native Matrix primitives.

Since the Raise-Hands feature would benefit a lot from being moderated we could start this journey first with a Reactions feature (👍️, 👏, 🤣, ❤️, ...). And here we could use the Matrix reactions feature or similar.

[mgcm]

Since the Raise-Hands feature would benefit a lot from being moderated we could start this journey first with a Reactions feature (👍️, 👏, 🤣, ❤️, ...). And here we could use the Matrix reactions feature or similar.

Thanks for the feedback!

I'm also interested in reactions but not message reactions like in MSC2677 - something more like what Jitsi and Teams consider to be reactions: playful animation overlays (+ sound?) that transmit a ephemeral feeling (indeed like 👍️, 👏, 🤣, ❤️ but for video).

You mention moderation, which I understand could apply to stuff such as muting, spotlighting and lowering hands for other users - I assume getting people off the call can use existing kick / ban membership events as spec'ed.

Under this light, here are a few proposals how this could be done:

Option 1 - A new moderation state event

Since a raised hand is a binary state (either a call member has its hand raised or doesn't), we could introduce a room state event (m.call.moderate) which would have a list of MXIDs which have their hand raised:

{
	"type": "m.call.moderate",
	"sender": "@alice:wonderland.org",
	"content": {
		"raised_hand": [
			"@whiterabbit:wonderland.org",
			"@redqueen:wonderland.org"
		]
	},
	"state_key": "",
	"event_id": "$143273582443PhrSn:example.org",
	"origin_server_ts": 1432735824653,
	"room_id": "!jEsUZKDJdhlrceRyVU:example.org",
	"unsigned": {
		"age": 1234
	}
}

Clients would add / remove their user's MXID to the list and use it to render accordingly. It could be further used to include other moderation features, like:

{
	"type": "m.call.moderate",
	"sender": "@alice:wonderland.org",
	"content": {
		"raised_hand": [
			"@whiterabbit:wonderland.org"
		],
		"hard_mute": [
			"@whiterabbit:wonderland.org",
			"@redqueen:wonderland.org",
			"@madhatter:wonderland.org"
		],
		"spotlight": [
			"@alice:wonderland.org"
		]
	},
	"state_key": "",
	"event_id": "$143273582443PhrSn:example.org",
	"origin_server_ts": 1432735824653,
	"room_id": "!jEsUZKDJdhlrceRyVU:example.org",
	"unsigned": {
		"age": 1234
	}
}

Having an empty state key, means any user with an adequate power level can update this state event. Room creation should make sure call members have the adequate power level to send m.call.moderate state events, so they can raise their hand.

Main issue with this approach is that anyone that is allowed to raise a hand is now also able to interfere with any of these moderation options, which is can be problematic.

Option 2 - Reactions to membership events

A raised hand could be an m.reaction room event that is linked to the m.room.member state event for the user:

{
    "type": "m.reaction",
    "sender": "@alice:wonderland.org",
    "content": {
        "m.relates_to": {
            "rel_type": "m.annotation",
            "event_id": "$event-id-of-m.room.member-for-alice",
            "key": "✋"
        }
    }
}

Un-raising your hand would be achieved with a redaction. With this approach, only the sending user and room moderators would be able to redact the raised hands.

Main issue with this approach is how to handle users that join the call after these reactions have been sent, potentially being unable to retrieve them from the room timeline?

Option 3 - A new dedicated room state event

Similar to the first option, but used only for raised hands:

{
	"type": "m.call.raised_hands",
	"sender": "@alice:wonderland.org",
	"content": {
		"@whiterabbit:wonderland.org": {
			"order": 1,
			"raised_at_ts": 1432735824653
		},
		"@redqueen:wonderland.org": {
			"order": 2,
			"raised_at_ts": 1432736824653
		}
	},
	"state_key": "",
	"event_id": "$143273582443PhrSn:example.org",
	"origin_server_ts": 1432735824653,
	"room_id": "!jEsUZKDJdhlrceRyVU:example.org",
	"unsigned": {
		"age": 1234
	}
}

This has the same downside of Option 1 (anyone can add or remove himself and others from the list) but how critical is messing up the intended raised hands state by malicious actors? Such manipulation would be easily handled by kicking / banning them.

Anyway, hope this can get the conversation one step further.

[amshakal]

Thank you for your work on this feature! I have a few design suggestions that could help simplify its implementation:

1. Emoji Usage: Instead of using a separate icon for the "Raise Hand" feature, could we consider using the hand emoji (✋) instead?
2. Integration with Reactions: Would it be possible to integrate the "Raise Hand" feature within the existing reactions menu, allowing users to use the hand emoji as a reaction?

I believe these changes could help streamline the user interface by reducing the number of options available in the bottom action sheet. A quick competitive analysis also showed that similar products often combine these features. What do you think?

Looking forward to your thoughts!

[mgcm]

@amshakal Great to hear your feedback. For the time being, my focus will be on the technical implementation of the feature and not worry much about the final UI/UX polish, as that is something that surely Element's Design team will have a strong opinon about.

[mgcm]

So I'm drafting an implementation using annotations and reactions (Option 2) and was wondering if the reaction should be to the m.room.member event or to the m.call.member event.

Having it linked to the room member state event would open this feature up to non-call use cases - I've read about people being interested in having raised hands in text chat rooms... but maybe thats too much forward thinking?

Either way, it's a minor implementation detail.

[CLAassistant]

All committers have signed the CLA.

[mgcm]

Updating to a refactored version that uses reactions to call membership events. It works :)

@amshakal I wasn't aware you were actually working for Element Design 🤦 I'm sure we can get something done along what you were suggesting.

I won't be focusing yet on the reactions feature but we can look at the raise hand emoji suggestion you mentioned. I'm fine with replacing the current SVG with the hand emoji (✋). Let's try that.

[mgcm]

Here's two screenshots of replacing with the hand emoji. I think this needs some tweaking on:

1. the bottom menu button active state: when the hand is raised, the contrast seems to be not optimal
2. the top left corner raised hand indicator per each call member: it's using the --cpd-color-icon-secondary as the background - should it be using another design token?

[toger5]

Main issue with this approach is how to handle users that join the call after these reactions have been sent, potentially being unable to retrieve them from the room timeline?

The relations api should solve this. joining a call gives you access to the room state and you can ask for the related raise hand events (even through the widget api)

[mgcm]

The relations api should solve this. joining a call gives you access to the room state and you can ask for the related raise hand events (even through the widget api)

@toger5 it works 👯

I've tested this in Widget mode inside Element Web and seems to work as well, although I'm wondering how as I don't see an implementation for relations in the RoomWidgetClient class from the JS SDK...

Also, on mobile, it loads and shows reactions from other non-mobile clients but sending isn't working.. wondering how to debug that...

[toger5]

I've tested this in Widget mode inside Element Web and seems to work as well, although I'm wondering how as I don't see an implementation for relations in the RoomWidgetClient class from the JS SDK...

That is surprising indeed. I also expected that to be a required change (to update the js-sdk matroska widget client to forward relations requests to the host client)
But the relations would only be needed on load right? So a widget still gets the up to date state by just subscribing to the raise hand room events?

Also, on mobile, it loads and shows reactions from other non-mobile clients but sending isn't working.. wondering how to debug that...

Probably you dont need to do too much debugging here. On mobile we are very strict with the permissions. The raise hand event most likely is just not part of that yet. I am wondering how you can receive them though.

[mgcm]

That is surprising indeed. I also expected that to be a required change (to update the js-sdk matroska widget client to forward relations requests to the host client) But the relations would only be needed on load right? So a widget still gets the up to date state by just subscribing to the raise hand room events?

Yes, relations are queried only on the call's session first load.

Probably you dont need to do too much debugging here. On mobile we are very strict with the permissions. The raise hand event most likely is just not part of that yet. I am wondering how you can receive them though.

🤔

[mgcm]

This is ugly. Could use some hints on how to improve the emoji rendering style.

[toger5]

On the topic of reactions we might want to explore lottie files. That is probably very little effort and just showing the lottie file over the tile could already look really good.
But for this the scope of raise hands is probably a better start.

[toger5]

@fkwp and I crossed this PR in a meeting and concluded the following:

• Redactions in widgets work differently than in the cs-api

  • The cs api uses a custom endpoint and does not allow (at least synapse does fail with an internal server error) sending an event of type m.room.redaction

  • The widget api does not have a custom action for redactions but uses a send action with a m.room.redaction

  • The react sdk widget driver special cases and sends the custom cs-api endpoint request if a send of type m.room.redaction is received:

    } else if (eventType === EventType.RoomRedaction) {
           // special case: extract the `redacts` property and call redact
           r = await client.redactEvent(roomId, content["redacts"]);
    }
    

  • All this allows us to redact events if the js-sdk matroska mode would translate client.redactEvent into a send event widget action.

• Overall if we want to merge sth that marks the first step into this direction reactions which time out and do not need to be redacted would make it much simpler and quicker to have a usable first version (thumbs up and a heard emoji for instance. If it is possible to build this with lotti emojis that would be really great)

I will try looking into adjusting the js-sdk matroska mode but there is still the issue that the rust sdk is not handling this special case.

[toger5]

All this allows us to redact events if the js-sdk matroska mode would translate client.redactEvent into a send event widget action.

This is not 100% correct. The js-sdk internally does convert events into redact PDU's internally and hence the widget api already gets the correct event for the encryptAndSendEvent function (that the matroska mode overwrites to make use the widget api instead)
I really wanted to find out why its still not working. It turned out that the only issue is, that the internal js-sdk redaction PDU has the redact property outside the content where the widget api expects it to be part of the content...

So its basically a bug in the matroska mode that is easy enough to fix:
matrix-org/matrix-js-sdk#4398

The only missing bit now is support in the rust widget driver.

Currently the rust sdk wont approve capabilities for the redactions and reactions.
@mgcm The lack of that capability should be used to hide the according buttons.

[toger5]

we can use

Suggested change

	setInitClientState({
	widgetApi: initClientState.widgetApi,
	client: initClientState.client,
	passwordlessUser: false,
	});
	setInitClientState({
	...initClientState,
	passwordlessUser: false,
	});

here.