Limitations of RmlUi data bindings

[exjam]

I am struggling with the RmlUi data binding model, and given the length of this write up I am really hoping I am not missing something obvious 🤞.

My misadventures in RmlUi data bindings.

Imagine you have a simple mail reading application, with a very traditional UI where you have a list box containing the list of emails in your inbox and another area to show the contents of the mail you are currently reading.

----------------------------------
| mail 1 | From: friend          |
| mail 2 | Sent: 01 / 01 / 1970  |
| mail 3 | Hello world!          |
| ...    |                       |
----------------------------------

So you might start with a data model backing this which is simple std::vector of Mail, e.g.:

struct Mail {
    uint64_t id;
    uint64_t timestamp;
    std::string from;
    std::string message;
    bool read;
};
std::vector<Mail> m_mail;

auto mail = constructor.RegisterStruct<Mail>();
mail.RegisterMember("id", &Mail::id);
mail.RegisterMember("timestamp", &Mail::timestamp);
mail.RegisterMember("from", &Mail::from);
mail.RegisterMember("message", &Mail::message);
constructor.RegisterArray<std::vector<Mail>>();
constructor.Bind("inbox", &m_mail);

Handling the html for the list of mail is very simple and we already have the data-for binding which is great for this!

<listbox id="inbox">
    <listbox-item data-for="mail, mail_index : inbox" data-attr-value="mail_index">
        {{mail.timestamp} {{mail.from}}
    </listbox-item>
</listbox>

Unfortunately handling the UI for the reading the mail is more complicated... we would want something like this:

From: {{currently_being_read_mail.from}}
Sent: {{currently_being_read_mail.timestamp}}
{{currently_being_read_mail.message}}

But, what is currently_being_read_mail here?

The first way I did this was to add a read_mail_index variable to my data model:

int m_read_mail_index;
constructor.Bind("read_mail_index", &m_read_mail_index);

<listbox id="inbox" data-attr-value="read_mail_index">...</listbox>

From: {{inbox[read_mail_index].from}}
Sent: {{inbox[read_mail_index].timestamp}}
{{inbox[read_mail_index].message}}

This trivial implementation gives one immediate problem: if your inbox is empty this will cause an error because there is no valid inbox[n] when your inbox is empty leading to an out-of-bounds error. My first thought was that, that is easy, we can use data-if and get rid of this UI when there is no mail:

<div data-if="read_mail_index < inbox.size">
   From: {{inbox[read_mail_index].from}}
   Sent: {{inbox[read_mail_index].timestamp}}
   {{inbox[read_mail_index].message}}
</div>

Except this does not work, as data-if does not remove elements from the DOM but merely sets display: None and the child data bindings are still being updated. So there is another very messy workaround you could do which is only possible thanks to the very recent PR#740:

<div data-if="read_mail_index < inbox.size">
   From: {{read_mail_index < inbox.size ? inbox[read_mail_index].from : ''}}
   Sent: {{read_mail_index < inbox.size ? inbox[read_mail_index].timestamp : ''}}
   {{read_mail_index < inbox.size ? inbox[read_mail_index].message : ''}}
</div>

This now technically works, but it is ugly and, more importantly, we have had to add something which is purely UI logic to our data model.

This can be seem as problematic from an ideological point of view of "separation of concerns", in that I would like all my UI code to be together in one place, but it is also problematic from a techincal perspective:

• What if I had multiple Mail reading windows? They would all need their own read_mail_index0, read_mail_index1, ... etc variable to use.
• What if the UI was completely user made (e.g. because you have modding support), they would not be able to implement this without edits to the C++ data model.
• Even if it was not user made, what if we used Lua scripting to drive our UI instead of C++? Our lua would be reliant on our C++ data model having something added to it.

OK so it might seem like you can solve all of those 3 above points by using Lua scripting (if you did not have the lua/user requirement you could have also done this same "solution" from C++). Let's say you fully integrated Lua + RmlUi into your code base, well, you'd be immediately stuck as there is no way to access the internal DataModel header - so let's say we modified RmlUi to expose the internal DataModel.h header which allowed us to provide proper access to the DataModel from Lua:

<listbox id="inbox" onchange="on_selected_mail_changed(tonumber(event.parameters['value']))">...</listbox>

<div>
   From: <span id="read-from"></span>
   Sent: <span id="read-sent"></span>
   <span id="read-message"></span>
</div>

<script>
data_model = context:GetDataModel("mail")

function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    read_mail_index = index
    if mail then
        element = document:GetElementById("read-from")
        element.inner_rml = mail["from"].value

        element = document:GetElementById("read-sent")
        element.inner_rml = mail["timestamp"].value

        element = document:GetElementById("read-message")
        element.inner_rml = mail["message"].value
    end
end
</script>

So then you go play around with your UI in your application and it all seems to be working, big success! But something happens and you soon remember why the whole data bindings thing even exist to begin, if the data changes this change will not be reflected in our programatically driven UI. There is no connection between model.DirtyVariable("inbox") and this Lua code to update the currently read mail. In my case it was simply that we were reading mail index 0 when a new mail came in, which updated the listbox's data-for but as the listbox's value did not change (we still had index 0 selected) it did not update the currently read mail, this meant it showed the contents of the previous mail at index 0 whilst the listbox showed that you had selected the new mail at index 0.

So then you might be like OK I got a real big brain and I can come up with a solution to this too! We could just change our Lua code to write data models but with the exact index in, this is like our first approach of {{inbox[read_mail_index].xxx}} but without any of the downsides of requiring UI logic variable in the data model and without the out-of-bounds access problems.

function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    if mail then
        element = document:GetElementById("read-from")
        element.inner_rml = "{{mail[" .. index .. "].from}}"

        element = document:GetElementById("read-sent")
        element.inner_rml = "{{mail[" .. index .. "].sent}}"

        element = document:GetElementById("read-message")
        element.inner_rml = "{{mail[" .. index .. "].message}}"
    end
end

So you excitedly open your game again and click around your inbox and it seems to be working, but you notice something weird. For the duration of 1 frame the text in your UI is the string of the data binding "{{mail[0].from}}" before it updates to "Friend". This seems strange so you set some breakpoints to realise once again you have been bitten by order-of-operations with regards to RmlUi's update loop (I say bitten once again because I have found this was a real hard learning point of RmlUi, it was difficult to get the order of operations correct when dealing with the ordering interactions between layout / data binding / data callbacks / event handling / rendering). You can see in the below code from Context::Update all the data bindings are updated, and then the Update function is called on your documents which will lead to things like onchange event handler being called and consequently your Lua code which updated the DOM, so the data binding you set will not be applied until the next frame when you call to Context::Update again.

bool Context::Update() {
...
   // Update all the data models before updating properties and layout.
   for (auto& data_model : data_models)
      data_model.second->Update(true);
...
   root->Update(density_independent_pixel_ratio, Vector2f(dimensions));

After feeling very defeated you pour over the RmlUi documentation looking for a ray of hope, any hack you can apply to get the result you want, and then you see data-alias. At first glance data-alias seems like it might help here, it would allows us to get code much closer to my first proposed data binding, and even though the alias lives in the data model it is scoped to an Rml::Element * which gives us the per-document UI logic semantics that we were looking for.

<div id="read-mail" data-alias-currently_being_read_mail="mail[read_mail_index]">
   From: {{currently_being_read_mail.from}}
   Sent: {{currently_being_read_mail.timestamp}}
   {{currently_being_read_mail.message}}
</div>

The problem we have is that as mentioned before we want read_mail_index to be a Lua variable and not a data binding, so you have a great idea, lets combine our previous ideas and just modify the data-alias with the index value from lua!

function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    if mail then
        element = document:GetElementById("read-mail")
        element:SetAttribute("data-alias", "mail[" .. index .. "]")
    end
end

So you start up your game and then... nothing happens... you step into the SetAttribute function and see that you cannot set or modify any data views from SetAttribute. So then you go on to look at how data-alias is stored in the data model and realise there is this nice method DataModel::InsertAlias which you could call to update what the alias points to, so you try it and then... nothing happens... turns out data-alias is immutable in that every data view and data expression resolves addresses at time of creation (which will resolve an alias to the address that the alias points to), so changing the alias will do nothing for currently existing DOM elements. And then thinking about it, even if data-alias was mutable, you would still need to bind this data-alias to something when the inbox is empty else you end up back with the original out-of-bounds access problem.

Enough complaining, what about solutions?

Well this is why I made an Issue instead of a Pull Request, I do not really know what an acceptable solution is which is why I am requesting feedback, I have two ideas though.

1. We could support multiple DataModel in one DOM hierarchy, I do not know if there is more history around why you're only allowed exactly one data model per DOM hierarchy other than the idea that it makes things much simpler (which is a totally worthwhile goal), but by having multiple data models then you could have a "UI logic" DataModel along side your actual data DataModel, and consequently your fully moddable Lua code could be able to create a data model and use it within the same UI that uses original data model from the game's data.

2. Mutable data-alias with nullable types, unfortunately data-alias is currently immutable in that data views resolves addresses at time of creation and data expressions at the time of parsing. If we were to allow data-alias to change at runtime then we would have to have a way to force everything to re-resolve their addresses and also to dirty their state. Additionally for mutable data-alias to be useful for the exact use case specified above we would need support for a null type, and as data-alias only supports DataAddress, then null would need to be a valid DataAddress to the null type. You can find an extremely hacked together mutable data-alias here

I am not sure if I had to pick exactly one of these two solutions which one I would go for. Multi-DataModel gives much more moddability support to our users in that they could mix their own lua data models in with our c++ authored game data models but it will come at the cost of much more complexity (e..g what happens if you have multiple data models active with the same name etc). Mutable data-alias seems like a less radical change to the existing data model structure to directly fix this problem, but you restrict moddability wrt mixing data models.

[exjam]

Another way of looking at the problem which someone just mentioned to me is, if we stick with the LUA editing DOM approach:

read_mail_index = 0
function on_selected_mail_changed(index)
    mail = data_model["inbox"][index]
    read_mail_index = index
    if mail then
        element = document:GetElementById("read-from")
        element.inner_rml = mail["from"].value

        element = document:GetElementById("read-sent")
        element.inner_rml = mail["timestamp"].value

        element = document:GetElementById("read-message")
        element.inner_rml = mail["message"].value
    end
end

Then what is missing is that I would want the document event on_selected_mail_changed(read_mail_index) to be called in the case of model.DirtyVariable("inbox").

So looking at it from that perspective, I could also solve this by adding a DataView implementation which calls a callback on update, though it would need to be attached to some element. This gives you the advantage of having C++ code or Lua code be able to observe and react to any changes in the data model.

You can find a hacked together implementation here.