implement VideoViz to record model runs in a video

[wang-boyu]

Summary

This PR consists of three commits:

1. Various draw_space() functions are moved and refactored into a separate matplotlib_renderer.py file.
   Previously:

   draw_space(grid, agent_portrayal, ax)

   Now:

   space_renderer = SpaceRenderMatplotlib(agent_portrayal=agent_portrayal)
   space_renderer.draw(model, ax)

   and for measures:

   measure_renderer = MeasureRendererMatplotlib("measure_name")
   measure_renderer.draw(model, ax)

   This is mainly to extract matplotlib plotting functions/classes out of Solara components, so that they can be re-used by other visualization methods, such as the VideoViz in the third commit. Besides, I don't think it changes how SolaraViz is used.

2. make_space_component and make_plot_component are moved from visualization.components.matplotlib into visualization.solara_viz, with a backend option to choose from matplotlib or altair. Default space drawing is changed from altair to matplotlib.

3. Implement a VideoViz that has similar APIs to SolaraViz with an example for the basic Schelling model. Essentially, running

   cd mesa/examples/basic/schelling
   python3 video.py

   will produce a video.mp4 file:

   schelling.mp4

   As mentioned in 1), VideoViz uses common matplotlib renderers extracted from SolaraViz.

Usage Examples

In file mesa/examples/basic/schelling/video.py.

[EwoutH]

I will look at the code tomorrow, but the video looks lit! 🔥

[quaquel]

I'll take a detailed look tomorrow. I like the video!

This does, however, interact with #2447, and I am, at face value, not yet convinced by the OO style over the function style API for now (which also interacts with some of the ideas in #2389). @wang-boyu, how do you view the interdependencies between this pr and #2447 and #2389?

[wang-boyu]

Oh sorry I didn't mean to replace #2447. I wanted to have VideoViz for a project I'm working on and ended up with this PR. To me the OO-style vs. functional style is really just a design choice -

1. It doesn't really affect how users use SolaraViz and make_space_component.
2. I chose OO-style to have a common protocol draw(model, ax) -> plt.Axes between space and measure renderers, so that VideoViz can have components: Sequence[MatplotlibRenderer]. But this should also be possible with the functional-style API.

So if we end up with funcitonal-style API instead, VideoViz can be built based on that as well. I think my main point here is to isolate matplotlib rendering out of SolaraViz so they it can be re-used by VideoViz.

Regarding #2389, I really like the ideas of having the more powerful APIs, instead of defining dictionaries in a agent_portrayal function. It reminds of me of Hadley Wickham's A Layered Grammar of Graphics and ggplot2 in R (which is also funtional style btw). I think what we want is something very similar but for ABM visualization? My concerns are that 1) designing such APIs takes lots of efforts, and 2) implementing the APIs also takes lots of efforts. But it would be truly great to have.

[quaquel]

Oh sorry I didn't mean to replace #2447. I wanted to have VideoViz for a project I'm working on and ended up with this PR.

No worries. Having VideoViz seems like a good idea, would close #885, and it's just very useful. I only want to make sure that we maintain some coherence in the development.

So, in my understanding, this pr adds:

1. Various draw_space() functions are moved into a separate matplotlib_renderer.py file.

This directly overlaps with #2447. It seems we agree that separating the draw_x functions into their own namespace makes sense. @wang-boyu, would you have a chance to review #2447 and let us know what you think? We either move forward there, or we move forward here with this code reorganization.

2. Refactoring the draw_x functions into an OO-style API.

I saw your comment in #2389 and hope to return to that asap. For MESA 3.0, I would counsel against this refactoring because it is breaking the established API. In my view, we should hash out an OO API for space and plot drawing, have this as experimental in 3.1, and see where it goes. I suggest we have the conversation on this in #2389. A major challenge, I expect, will be to abstract away the backend, given how radically different Altair and Matplotlib are.

3. make_space_component and make_plot_component are moved from visualization.components.matplotlib into visualization.solara_viz, with a backend option to choose from matplotlib or altair.

This directly overlaps with #2447, although I moved these functions into the __init__ of the components namespace. To me, it makes more sense to keep everything component related in the component namespace. They would, however, still be importable from mesa.visualization, so from mesa.visualization import make_space_component.

4. Default space drawing is changed from altair to matplotlib.

Yes, I agree, and this was also on my to-do list for after #2447.

5. Implement a VideoViz that has similar APIs to SolaraViz

The core of this PR, in my understanding, and, yes, a very good idea.

So, I propose the following next steps.

1. @wang-boyu could you share your thoughts on api reorganization #2447? I would like to come to a decision on whether to do the API reorganization via that PR or this PR.
2. Open a separate small PR, only changing the backend of solara_vis from Altair to matplotlib. This is a good idea and can be its own separate atomic PR.
3. how hard would it be to modify VideoViz to use the draw_x / make_x_component? That would be, for now, my preferred implementation that could even be included in MESA 3.0 if others agree.

[quaquel]

great to see the examples used for this as well.

One concern, not for this pr, is how to add these additional files to the docs.

[quaquel]

In the other parts of the visualization, we shifted to just calling it a plot component; what are the reasons for using the, in my view, ambiguous label of measure here?

[wang-boyu]

Thanks for asking. I kept the impression from the old (now experimental) SolaraViz:

mesa/mesa/experimental/solara_viz.py

Line 58 in 69571b4

if "Space" in layout_type:

and

mesa/mesa/experimental/solara_viz.py

Line 72 in 69571b4

elif "Measure" in layout_type:

where there's a card titled "Space" and another titled "Measure". "Plot" seems a bit amgiguous for me because we can have a plot for space, a plot for measure, and possible plots for other stuff. I might have missed some discussions when we made the change.

[quaquel]

I just followed a suggestion from @Corvince in #2443.

I agree on the ambiguity, but I contend that space is different in kind from, say, line plots, histograms, or other visualizations of data created by a model. So, for now, I am fine with separeting space from other kinds of plots.
Since we are still hashing out and refining the visualization style, I am fine with this distinction at the moment.

[quaquel]

This is perhaps for users the most tricky part. I agree on the error handling here. I would suggest also adding this to the module level docstring.

[quaquel]

I suggest making clear that this only works with the matplotlib backend

[EwoutH]

I will give you two a bit of time to align on this topic, let me know if a fresh perspective is appreciated!

[wang-boyu]

Thanks for reviewing this PR @quaquel. Appreciate it. I'll respond in next few days and also take a look at #2447. Lots of changes are indeed similar despite in different styles.

[Corvince]

I thought I already commented on this PR, but it seems I only wanted to do so. Nice work with the output, the video looks very clearn.

On the topic of OO vs FP: There are use-cased for both approaches, but I don't think OO is very beneficial in this case. The only available method is "record", so why not use it directly?

#Instead of 
viz = VideoViz(
    model,
    components,
    title="Schelling's Segregation Model",
)
viz.record(steps=50, filepath="schelling.mp4")

# Just do
record(
    model, 
    components, 
    title="Schelling's Segregation Model",
    steps=50,
    filepath="schelling,mp4
)

This leaves more room for users to compose how they want. To replicate your interface

def my_record(steps, filepath):
    return record(
        model, 
        components, 
        title="Schelling's Segregation Model",
        steps,
        filepath
    )

my_record(50, "schelling.mp4")

Or maybe you only want to change the model

def my_record(model):
    return record(
        model=model, 
        components=components, 
        title=f"${model.name} Segregation Model",
        steps=50,
        filepath=f"${model.name}.mp4"
    )

[wang-boyu]

Now that #2447 is merged, I'll update this PR accordingly.

[wang-boyu]

Thanks @Corvince for your feedback. I mainly wanted to have similar APIs in creating the different Viz "frontends":

page = SolaraViz(
    model,
    [make_space_component(agent_portrayal=agent_portrayal),
     make_measure_component("happy")],
)

viz = VideoViz(
    model,
    [make_space_component(agent_portrayal=agent_portrayal),
     make_measure_component("happy")],
)

where the usages might differ:

page   # simply use the page to render Solara webpage

viz.record(steps=50, filepath="schelling.mp4")

[wang-boyu]

Rebased to main branch and updated for #2447:

1. Extract a draw_plot method similar to draw_space for plotting measures
2. Rename mpl_space_drawing.py to mpl_drawing.py since it now contains both draw_space and draw_plot
3. Update the implementation of VideoViz to use draw_space and draw_plot

[quaquel]

I looked through the code and have at the moment a few high-level reactions

1. Extract a draw_plot method similar to draw_space for plotting measures

I don't mind this change, but I am not fully convinced it is needed.

2. Rename mpl_space_drawing.py to mpl_drawing.py since it now contains both draw_space and draw_plot

Given 1. this makes perfect sense.

3. Update the implementation of VideoViz to use draw_space and draw_plot

Here I have some questions. On the Solara side, we call things components because that is what they are in Solara. You borrow that terminology here, even though this code, as far as I can tell, will only work with matplotlib. So, does this component naming make sense, given that VideoViz only works with matplotlib?

In my understanding of the API based on the typing information, a component is a callable that is called with the model and the ax as arguments. But do you need the complexity of videoviz specific make_plot_component and make_space_component? Looking at your code, there is some convenience in both functions, so there might be a good argument for keeping them. However, is calling it component the best choice then? Effectively these make_ functions wrap a draw function that takes the model and the ax as input.

I understand your motivation for going OO to keep the similarity with SolaraViz. I don't mind doing it that way, but again, I wonder whether just accepting that it will be matplotlib specific might not open up a more generic and powerful API.

For example, you hard-code the grid layout at the moment. What if the user wants to change this? For example, in a 16:9 aspect ratio slide, I might like to have a 3x2 grid, while on a web page, I might prefer 2x3. What about having a gridspec optional keyword argument on the __init__?

To be clear, I needed to make a short mp4 of a mesa model today, and having this in would have saved me quite some time. I am just not yet fully convinced about some of the design choices made here.

[wang-boyu]

Thanks for the comments!

1. Extract a draw_plot method similar to draw_space for plotting measures

I don't mind this change, but I am not fully convinced it is needed.

If we want the same plot for measures in SolaraViz and VideoViz, without extracting a common method draw_plot, then there'll be two plotting functions that are essentially the same...?

3. Update the implementation of VideoViz to use draw_space and draw_plot

Here I have some questions. On the Solara side, we call things components because that is what they are in Solara. You borrow that terminology here, even though this code, as far as I can tell, will only work with matplotlib. So, does this component naming make sense, given that VideoViz only works with matplotlib?

I'm hoping there could be more backend options for VideoViz in the future, other than matplotlib. It'll be easier to develop and integrate new backend into both SolaraViz and VideoViz with the current API design, hopefully.

In my understanding of the API based on the typing information, a component is a callable that is called with the model and the ax as arguments. But do you need the complexity of videoviz specific make_plot_component and make_space_component? Looking at your code, there is some convenience in both functions, so there might be a good argument for keeping them. However, is calling it component the best choice then? Effectively these make_ functions wrap a draw function that takes the model and the ax as input.

Yes, these make_ functions took the ideas from make_mpl_space_component and make_mpl_plot_component that are also wrappers of functions that take a model and return a Solara component. They differ in how they are created - space needs portrayal functions, and plot needs measure names.

For example, you hard-code the grid layout at the moment. What if the user wants to change this? For example, in a 16:9 aspect ratio slide, I might like to have a 3x2 grid, while on a web page, I might prefer 2x3. What about having a gridspec optional keyword argument on the __init__?

Yeah it'll be good to allow users to configure grid layout, something I also wondered here: #2236 (reply in thread). I would prefer some common ways of addressing this, and a common API between SolaraViz and VideoViz would benefit from it, e.g.:

# `make_` functions are imported from mesa.visualization
layout = SomeLayoutCustomization(
        make_space_component(agent_portrayal=agent_portrayal),
        make_measure_component("happy")
)

page = SolaraViz(model, layout)

# here `make_` functions are imported from mesa.visualization.video_viz
layout = SomeLayoutCustomization(
        make_space_component(agent_portrayal=agent_portrayal),
        make_measure_component("happy")
)
viz = VideoViz(model, layout)

so that when users replace SolaraViz with VideoViz, they can expect the same layout to be exported in a video.

I'm also wondering whether it'll be possible to make the make_ functions even more generic - return Solara components when they are used in SolaraViz, and return video components when used in VideoViz. This way users don't need to change where they are imported.

Or maybe something even further:

model_explorer = ModelExplorer(
    model,
    SomeLayoutCustomization(
        make_space_component(agent_portrayal=agent_portrayal),
        make_measure_component("happy")
    ),
)

# renders Solara app if used in Jupyter notebook or called in `solara run app.py`
model_explorer

# exports a video
model_explorer.record(steps=10, filepath="my_amazing_model.mp4")

I guess this is beyond the scope of this PR? Apologies for being off topic. But this is also close to what @Corvince proposed above, to have just a single record() function.

[quaquel]

Thanks for the extensive reaction.

If we want the same plot for measures in SolaraViz and VideoViz

This is the core of the discussion. Do we want to make it trivial to swap between the UI and recording a movie? I see great value in this for ease of use and API simplicity. So, ideally, all you would have to change is VideoViz to SolaraViz, while the rest stays the same. Or indeed, as @wang-boyu describes at the end, you might even collapse it all into a single ModelExplorer class. While this is an ideal to aim at, I am not convinced that it is achievable in part because of the second point.

I'm hoping there could be more backend options for VideoViz in the future, other than matplotlib.

Do we have any description of creating mp4's with altair or plotly (a possible other backend supported in solara)? If we have, then I am less sceptical about the potential of having VideoViz and SolaraViz working similarly. However, a quick search did not turn up anything for Altair.

If we don't have any clear direction for supporting both Altair and matplotlib in VideoViz, it might actually be confusing to a user to have the same API for both VideoViz and SolaraViz even though the former only supports a single backend. Also, for those familiar with matplotlib, which are most scientific users, I assume the entire API might become too restrictive.

Playing devil's advocate, another separate reason for not bothering with the same API for VideoVizandSolaraViz` is that you can do a simple screen recording of the UI. So, what is the value?

[wang-boyu]

you can do a simple screen recording of the UI

One motivation for me is that I will need to run the same model with many different parameter settings, and I don't want to do manual recordings for all of them.

Back on the APIs, I think the benefit of current design is that it is open for future backends, without making breaking changes.

What are the proposed changes to the APIs? I'm a bit lost in these discussions : ) Is it to change to a single record() function (still with the make_x_component functions) as suggested above?

[quaquel]

One motivation for me is that I will need to run the same model with many different parameter settings, and I don't want to do manual recordings for all of them.

Fair point and good motivation

Back on the APIs, I think the benefit of current design is that it is open for future backends, without making breaking changes.

Ok, but I still would like to understand the plausibility of this being realized in the future. It is great to design for future extensions, but if those future extensions are not plausible or the API becomes convoluted, why bother?

What are the proposed changes to the APIs? I'm a bit lost in these discussions : ) Is it to change to a single record() function (still with the make_x_component functions) as suggested above?

I haven't discussed or commented on the actual API in much detail yet. For me, the second point here is the key. If there is a plausible path to mp4 with Altair, I would comment on this API differently compared to a situation where the focus is on matplotlib only.

(and sorry for being perhaps a bit annoying, I really like the functionality provided here, but since it is new functionality, I believe it is worth it to have a slightly longer back and forth on how to best provide this functionality).