Tempe: Efficient Purely-Micropython Graphics

[corranwebster]

A couple of months ago I got ahold of a Pimoroni 320x240 ST7789-based LCD display. Since a lot of my work over the past 20 years has been around data visualization and UIs (I'm a long-time contributor to the Chaco plotting library for Python), I wanted to use it to display pretty plots but immediately ran into memory issues: basically a 320x240 16-bit framebuffer is too big if you are doing anything interesting with data. There are also issues with speed when you are trying to update dynamic visualisations which need to draw the entire buffer and transfer it to the display while remaining reasonably responsive.

These issues might be solvable by reducing to 8-bit color or using a smaller part of the screen, but it should be possible to use the full capabilities of these screens even if you can't have a full, in-memory buffer.

While there are C-based libraries which address this (in particular PicoVector looks interesting, and lvgl looks comprehensive) it seems like there is a place for a purely Micropython-based solution: something that you can just mip-install on top of standard Micropython and get to work.

Tempe is what grew out of this need.

Documentation

Even though it is in early stage development, Tempe has reasonably comprehensive documentation.

Tempe Documentation

• User Guide
  • Tutorial
• API docs

Pretty Pictures

Installation

Either

>>> mip.install("github:unital/tempe/src/tempe/package.json")

or

mpremote mip install github:unital/tempe/src/tempe/package.json

should work.

Only tested on the most recent Micropython, but should work on anything reasonably recent as long as it has framebuf.

Features

Based On framebuf

Tempe uses the standard framebuf module to do all rendering to bitmaps. This is a compromise of universality and speed vs. the somewhat limited API of framebuf, but it works well enough and allows the library to be written entirely in Micropython. Drawing heavily uses memoryviews and strided frame buffers to allow clipping drawing to certain regions.

Memory Efficient Drawing

Tempe transparently handles:

• tiled rendering to the screen when the available memory is smaller than the area to be rendered
• damage-region tracking, so that ideally only things that change need to be updated in the display

The basic object for drawing is a layered Surface which tracks objects and knows how to render them efficiently and tracks changes. You then give Tempe as large a buffer as you can afford, and tell it to refresh the display whenever you want to update things:

surface = Surface()

# draw things on the surface's layers
surface.rectangles("BACKGROUND", (0, 0, 320, 240), "#eee")
greeting = surface.text("DRAWING", (10, 10), "Hello world!", "#333")

# render to the display
surface.refresh(display, buffer)

# draw more things, or update old things
surface.circle("DRAWING", (50, 50, 20), "#ff0")
greeting.update(texts=["Hello Tempe!"])

# update only what has changed on the display
surface.refresh(display, buffer)

Optimised for Data Visualization

Tempe isn't a full plotting library, but is intended to be usable as part of one: for people familiar with the Grammar of Graphics it's concerned with rendering geometry and aesthetics, but doesn't have any particular support for statistics, scales, etc. Because of this intended use, the API is a bit different from standard drawing libraries: data visualisations usually draw many similar shapes (lines, rectangles, markers, etc.) but with different parameters (coordinates, colours, etc.), so the fundamental units of drawing are groups of similar objects.

For example, a horizontal bar plot is essentially a group of rectangles, which Tempe could create in one command as:

surface.rectangles(
    layer="DRAWING",
    geometry=[(4, 4 + 4*i, value, 3) for i, value in enumerate(data)],
    colors="#f00",
)

Memory Efficient Geometry

Because Tempe is particularly concerned with memory efficiency, it has ways of more efficiently representing geometries with repeated and predictable values, so the geometry of the rectangles could instead be represented as:

geometry=ColumnGeometry([Repeat(4), Count(4, 4), data, Repeat(3)])

which avoids storing many repeated values in lists or arrays.

Bitmap Fonts

Tempe can use either FontToPy or MicroFont fonts to draw nicer text than the default framebuf font.

Polar Coordinates

Tempe provides functions that convert geometry expressed in (r, theta) polar coordinates to standard (x, y) cartesian coordinates, making it straightforward to create polar line plots, radar plots, coxcomb plots, donut plots, and even pie charts (if you have to).

Asyncio Support

Tempe Surface objects have an asyncio Event that is triggered when a refresh is needed, so automatic updating for asyncio applications is as simple as:

async def refresh_display(surface, display, working_buffer):
    while True:
        await surface.refresh_needed.wait()
        surface.refresh(display, working_buffer)

Where is this all going?

I'd eventually like to have a Micropython plotting library that looks as good as Matplotlib and is reasonably responsive when updated, and Tempe is I think a step in that direction. Basing drawing on FrameBuffer is a bit limiting: in particular it would be nice to have options for sub-pixel rendering and alpha compositing - the latter in particular that may be possible with viper-acceleration.

Whether Tempe grows these features, or whether something us built on top of it I'm not sure.

How can you help?

This is an early release using hardware that I have access to and solving my problems. If this seems useful to you, please try it and at least give feedback on any rough edges or missing features that you'd like to see.

[peterhinch]

This looks impressive! Very pretty visualisations. I do have a query. You state

basically a 320x240 16-bit framebuffer is too big

a statement with which I agree wholeheartedly. Yet the library is based on framebuffer. How do you resolve this? (I've not found your ST7789 driver).

[corranwebster]

Yet the library is based on framebuffer. How do you resolve this? (I've not found your ST7789 driver).

The ST7789 driver is here: https://github.com/unital/tempe/blob/main/examples/devices/st7789.py - it's a bit sketchy but the key code there is the ability to blit into a windowed rectangle on the display (it's currently doing one row at a time in the window while I chase down what might be a bug with the SPI code if I try to give it a memoryview, so it's potentially much faster if I can fix that).

The other key pieces of code are the Raster object, which essentially creates FrameBuffers which are windowed memoryviews into other buffers using offsets and strides (allowing clipped drawing) and the refresh method of the Surface object, which does ad-hoc tiling if the damaged region is larger than the available working memory (I've been using 1/3 or 1/4 of the screen size for this and it seems to work reasonably unless you go crazy with fonts): https://github.com/unital/tempe/blob/fe63d247e477309d4c53775f661d6bc0e440949c/src/tempe/surface.py#L41-L49

        for rect in self._damage:
            x, y, w, h = rect
            # handle buffer too small
            buffer_rows = (len(working_buffer) // (2 * w)) - 1
            for start_row in range(0, h, buffer_rows):    # <--- this splits into tiles
                raster_rows = min(buffer_rows, h - start_row)
                raster = Raster(working_buffer, x, y + start_row, w, raster_rows)   # <--- this creates the framebuffer on top of the preallocated memory
                self.draw(raster)  # <--- this draws objects into the framebuffer, offset by the right number of rows and columns
                display.blit(working_buffer, x, y + start_row, w, raster_rows)  # <--- this transfers the drawn bytes to the correct rows and columns of the display

These sorts of tricks are common when using things like NumPy and Scikit-Image in regular Python (but we don't have NumPy's nice syntactical sugar!).

The downside of this approach is that objects that span multiple tiles get drawn by the framebuffer code multiple times at different offsets, but most of the framebuffer routines scale linearly with the number of rows rendered and you don't pay a huge penalty for drawing things outside of the current framebuffer's drawable region.

This is all very dependent on the ability of the display device to do updates to a small region of the screen - I know that's not true for all devices because I have a 7-color e-ink screen that has to update the entire screen in one go - but I suspect that this is true for a lot of these small 16-bit LCD/OLED screens.

[peterhinch]

Thanks for that, the penny has dropped :) Your use of FrameBuffer is ingenious, and not one I'd considered. Your docs are excellent.

You may be familiar with the display drivers for nano-gui and other GUI's. You and I approach the problem of excessive buffer size in different ways: I use a 4-bit color palette with a full screen frame buffer while you use multiple small RGB565 frame buffers blitted to the hardware. I guess that a data visualisation application can free the RAM used by the buffer once it has been blitted, however there could be an issue with RAM fragmentation if you regularly create and release frame buffers. In the GUI applications the one large buffer is instantiated very early and left in place, with drawing done using framebuffer primitives.

Perhaps you could clarify the allocation situation. Take the simple case of a text value which is periodically updated. Is a buffer created and destroyed on each update?

As a lazy sod, a benefit of my approach is that the display driver is very simple: creating a compatible driver for a new display is straightforward even when the hardware is quite different (e.g. ePaper).

Nitpicks

One easy byte saver is as follows. Where you have

GCTRL = const(b"\xB7")

the compiler allocates storage for GCTRL (Python naming rules). Space is saved by prepending the name with _:

_GCTRL = const(b"\xB7")

No storage is allocated, the symbol is used only at compile time.

Another nitpick. The docs have

# a buffer one quarter the size of the screen
working_buffer = array('H', bytearray(2*320*61))

This allocates a buffer of 19520 bytes. Should this read (e.g.)

# a buffer one quarter the size of the screen
working_buffer = array('H', (0 for _ in range(2*320*61)))

which allocates 39_040 bytes (1/4 the size of the screen)? Either the comment or the code seems wrong.

[corranwebster]

Thanks for the comments. I'd looked at nanogui but haven't tried it - unfortunately restricting to lower color resolutions and using a palette doesn't work when you want to use continuous colormaps like Viridis for data visualisations - they quickly eat up your available colors, even if you use a small subset of the full range. I think you have a distinctly different goal with nano-gui which is to support even minimal displays on extremely constrained devices, where with Tempe (at least for now) I am really only considering support for 16-bit displays and trying to access their full capabilities.

Perhaps you could clarify the allocation situation. Take the simple case of a text value which is periodically updated. Is a buffer created and destroyed on each update?

tldr: no - I allocate once and use the one buffer for (almost) everything.

The strategy that I've been using is the one you were describing to avoid fragmentation: allocate a large chunk of memory almost immediately and hold onto it. However unlike a standard screen buffer it's not the size of the full screen and I don't try to preserve the contents between writes to the display: it's not a cache, it's just scratch-space. I use the same memory for big writes and small writes: small ones just use the first part of the chunk of memory, while large ones repeatedly fill it to draw

So concretely what's happening this example (which is exactly the update some text periodically case): https://github.com/unital/tempe/blob/main/examples/update_example.py

• after initial imports, we allocate a large chunk of memory (in this example we have enough space for it to be 1/3 of the screen), so about 51200 or 25600 2-byte pixels (I actually allocate one line more because of this issue)
• we also import the PyToFont-style fonts early, again to avoid fragmentation, because this example uses big numbers
• the surface is initialised with a background rectangle and some empty text objects
• this marks the entire screen as needing a refresh and sets an asyncio Event
• we go into the asyncio loop:
  • the refresh task will immediately see that the screen needs a refresh, and so it calls the refresh method
  • the refresh task uses the allocated working memory as a framebuffer (each of the following uses most of the allocated memory):
    • draws the top third of the screen using framebuffer commands and transmits to the device
    • draws the middle third of the screen using framebuffer commands and transmits to the device
    • draws the bottom third of the screen using framebuffer commands and transmits to the device
    • resets the Event
  • periodically the other tasks poll the hardware, and when the value changes they update the text object
    • the update marks the text's area of the screen as damaged (in the example I've used an explicit clipping rectangles like (10, 10, 240, 40)) and sets the refresh needed Event
    • the refresh task can now wake up, since the event is set
    • the refresh task uses a view of only the first (240 * 40) = 9600 pixels (or 19200 bytes) of the allocated working memory as a framebuffer
    • because everything fits into the working memory, we only draw once using framebuffer commands (if the update had been bigger it would split into multiple draws as needed)
    • the contents of the buffer are transmitted to the display into just the area that needs updating (eg. starting at row 10, column 10, for 240 columns and 40 rows)
    • the Event is reset and the refresh task waits for the next change

As a lazy sod, a benefit of my approach is that the display driver is very simple: creating a compatible driver for a new display is straightforward even when the hardware is quite different (e.g. ePaper).

I'm lazy too! I'm sort of hoping that this code can sit on top of other people's display drivers as long as the display is capable of partial updates and the driver exposes that. For example, this ST7789 driver exposes a blit_buffer method: https://github.com/russhughes/st7789py_mpy/blob/7265925bd0c092e8105200d18b2dba9dfbc12c27/lib/st7789py.py#L485-L497 (I would have used that as a basis, but I couldn't get it to work with my hardware). So implementing a Tempe Display on top of that would look something like:

class AnotherST7789(Display):

    def __init__(self, driver):
        self.driver = driver

    def blit(self, buffer, x, y, w, h):
        self.driver.blit_buffer(buffer, x, y, w, h)

I really should add something about writing display classes to the documentation.

Space is saved by prepending the name with _:

Thanks! I didn't know this.

Another nitpick. The docs have

# a buffer one quarter the size of the screen
working_buffer = array('H', bytearray(2*320*61))

This allocates a buffer of 19520 bytes. Should this read (e.g.)

I think bytearray(2*320*61) allocates a buffer of 39040 read/write bytes, which the array then views as 19520 unsigned 16-bit integers (ie. RGB565 pixels) because of the H. This is (roughly) one quarter of the 76800 pixels of the screen.

I think this is faster than allocating using a generator (particularly when I don't care about the initial values in the array) since it avoids a large Python for-loop. I would be interested to know if there is a faster way to initialise an array of 16-bit values where I don't care about the contents - in CPython I'd normally just use numpy.empty(...) for this sort of thing.

[picpic020960]

Bonjour ,
It works also on micropython unix port ?
Widget as lvgl : button , slider ... ?

[corranwebster]

It runs on the unix port (I run the test suite on unix) but:

• you'll need some way to display on a screen - if you have a way of displaying a standard micropython framebuf.FrameBuffer you can probably make it work, but you'll need to write your own display driver
• unix systems typically have a lot more memory available than microcontrollers and are faster, so there is less need for the tricks that Tempe is using to draw efficiently
  If I knew the sort of screen you want to display on in with a unix system, I might be able to give more pointers.

There's no concept of widgets in Tempe - it's just a drawing library, unlike lgvl which is a full GUI system. You could draw buttons, sliders etc. using Tempe, but then you'd need to write the code to handle user-interactions (eg. button presses, etc.) from scratch.

[rkompass]

@corranwebster: Very impressing library, Thank you.
Do you have timings of rendering the graphics examples, say on a Pico?
(Or did I miss those?)
Another question:
Do you think that Tempe would benefit from asynchronous SPI? If yes: Roughly to what degree?

[corranwebster]

So on an original Raspberry Pi Pico running 125 MHz, this example:

takes about ~50 ms to draw initially and ~15 ms to re-draw when one of the values changes (which is faster than an update would take using a full-screen framebuffer).

This example:

has plots which were optimised for fast display and live updates and I was getting ~50 ms update times.

The fancy plots on the original post take about half a second to draw the first time, but they are doing a lot. Those aren't live-plotting examples, so I don't have timings for updates, but I would expect updates to be much faster because only what has changed gets redrawn. That's the drawing and SPI transmission time, not including the time to do data transformation to set up the plots. I think I could speed them up significantly with a little work (in particular, I'm letting most of the objects determine their bounds, rather than telling them what region they are drawing in).

In general, speed is heavily dependent on how the plots are set up and the amount of data, as well as screen layout. If you can align objects so that they fit entirely within your working memory, updates will be much faster, and if they don't cross tile boundaries initial drawing will be faster.

The general goals I've been aiming for are that it should feel responsive to a user:

• less than a second for a complex initial render, ideally less than 500 ms
• less than 100ms for an update, ideally less than 50 ms
  Since I've been within these goals, I haven't tried optimising hard yet. I might also need to revise them once I get some examples going with actual user interaction, but from my experience 100ms is fine for everything except mouse drag-style interactions.

So there are limitations - you're not going to be able to do video with this, for example, and the rest of the application has to be OK with pausing for however long it takes to update the screen. But for everyday UI purposes I expect it'll be fine.

In terms of async SPI I think that the answer is a big "it depends" - rough calculation is that it takes ~20ms to transmit a full 320x240 frame so for big initial plots you don't gain a lot if you can transmit in parallel, but there might be more of a win for updates. However asynchronous SPI I think means you need at least two buffers: the one being transmitted and the one being written, and splitting your buffer in half may end up costing more than you save from async SPI.

I would be interested in any async SPI library that's available - I don't recall seeing one, but I'm fairly new to Micropython.

[corranwebster]

Quick follow-up: thanks in part to the discussion below, I've found a bug that when fixed speeds up most of the initial renders by a factor of 2 on the drawing side, so the updating text example now has an initial draw time of about 36ms, and the complex plots are now taking about 250 ms.

[rkompass]

Nice.

[peterhinch]

@corranwebster Thank you for that detailed explanation. You've evidently given this a lot of thought.

I think you have a distinctly different goal with nano-gui which is to support even minimal displays on extremely constrained devices

That is true. As hardware improved I expanded the concept to full GUI's with pushbutton or touch input, and extended support to larger displays. The latter involved the compromise which is a 4-bit color palette. This works well with the widgets, but imposes limits on what you can display.

Re

working_buffer = array('H', bytearray(2*320*61))

Your observations on buffer size are correct. Age-related brain-fade on my part. However I'm unconvinced by

I think this is faster than allocating using a generator (particularly when I don't care about the initial values in the array) since it avoids a large Python for-loop.

My understanding is that a few years ago there was a major change in the way Python (and MP) handle generator expressions. I think they are now fast and memory efficient. I produced a file containing

from array import array
a = array("I", (0 for x in range(100)))

This produced the following bytecode

$ upython -v -v rats31.py
File rats31.py, code block '<module>' (descriptor: 7d0e70a06300, bytecode @7d0e70a065e0 38 bytes)
Raw bytecode (code_info_size=4, bytecode_size=34):
 20 04 01 2c 80 10 02 2a 01 1b 02 1c 02 16 02 59
 11 02 10 03 32 00 11 05 22 80 64 34 01 5e 34 01
 34 02 16 06 51 63
arg names:
(N_STATE 5)
(N_EXC_STACK 0)
  bc=0 line=1
  bc=12 line=2
00 LOAD_CONST_SMALL_INT 0
01 LOAD_CONST_STRING 'array'
03 BUILD_TUPLE 1
05 IMPORT_NAME 'array'
07 IMPORT_FROM 'array'
09 STORE_NAME array
11 POP_TOP
12 LOAD_NAME array
14 LOAD_CONST_STRING 'I'
16 MAKE_FUNCTION 7d0e70a06260
18 LOAD_NAME range
20 LOAD_CONST_SMALL_INT 100
23 CALL_FUNCTION n=1 nkw=0
25 GET_ITER
26 CALL_FUNCTION n=1 nkw=0
28 CALL_FUNCTION n=2 nkw=0
30 STORE_NAME a
32 LOAD_CONST_NONE
33 RETURN_VALUE
File rats31.py, code block '<genexpr>' (descriptor: 7d0e70a06260, bytecode @7d0e70a066a0 20 bytes)
Raw bytecode (code_info_size=6, bytecode_size=14):
 b1 40 06 04 07 20 53 b0 53 53 4b 06 c1 80 67 59
 42 38 51 63
arg names: *
(N_STATE 7)
(N_EXC_STACK 0)
  bc=0 line=1
  bc=0 line=2
00 LOAD_NULL
01 LOAD_FAST 0
02 LOAD_NULL
03 LOAD_NULL
04 FOR_ITER 12
06 STORE_FAST 1
07 LOAD_CONST_SMALL_INT 0
08 YIELD_VALUE
09 POP_TOP
10 JUMP 4
12 LOAD_CONST_NONE
13 RETURN_VALUE
mem: total=4725, current=1856, peak=2915
stack: 288 out of 80000
GC: total: 2072832, used: 2048, free: 2070784
 No. of 1-blocks: 20, 2-blocks: 5, max blk sz: 13, max free sz: 64700

[corranwebster]

@peterhinch So this got me experimenting and digging into the Micropython source to see just exactly what is going on. And yes, my mental picture was wrong.

Using array('H', bytearray(2*n)) is very fast - in my tests it takes about 5 ms for a 320x120 array but, digging into what Micropython is actually doing:

micropython/py/objarray.c

Lines 120 to 133 in d34b15a

	if (((MICROPY_PY_BUILTINS_BYTEARRAY
	&& typecode == BYTEARRAY_TYPECODE)
	|| (MICROPY_PY_ARRAY
	&& (mp_obj_is_type(initializer, &mp_type_bytes)
	|| (MICROPY_PY_BUILTINS_BYTEARRAY && mp_obj_is_type(initializer, &mp_type_bytearray)))))
	&& mp_get_buffer(initializer, &bufinfo, MP_BUFFER_READ)) {
	// construct array from raw bytes
	// we round-down the len to make it a multiple of sz (CPython raises error)
	size_t sz = mp_binary_get_size('@', typecode, NULL);
	size_t len = bufinfo.len / sz;
	mp_obj_array_t *o = array_new(typecode, len);
	memcpy(o->items, bufinfo.buf, len * sz);
	return MP_OBJ_FROM_PTR(o);
	}

It's doing a memcpy, so at best we can get half the available memory and memory is now fragmented. I can't use it to allocate a 320x240 array, for example. My mental model was that it was sharing the memory with the bytearray.

Using array('H', (0 for _ in range(320*120))) is slow - about 2 seconds in my tests. I think that's because the generator has no __len__ and so Micropython has to build the array incrementally:

micropython/py/objarray.c

Lines 150 to 151 in d34b15a

	if (len == 0) {
	array_append(MP_OBJ_FROM_PTR(array), item);

On the other hand, it can create a 320x240 array. I think that there is quadratic behaviour in there because it takes 8 seconds for an array 2x larger.

If you give it an iterator with a length (something like this):

class Allocator:
    
    def __init__(self, n):
        self.n = n
        self.count = 0
        
    def __len__(self):
        return self.n
    
    def __iter__(self):
        return self
    
    def __next__(self):
        while self.count < self.n:
            self.count += 1
            return 0
        else:
            raise StopIteration()

it is twice as fast because it can allocate the memory once and then iterate through the values.

But given that I don't care about the initial values, array('H', range(320*120)) is significantly faster: about 100 ms.

All that said, I think this means I need to go back and look over my code and see if I really need arrays of 16 bit ints or whether I can get away with working with bytearrays.

So thank you for the feedback!

[peterhinch]

Thanks for that, thought-provoking. Those timings are seriously slow. It's not obvious why allocating an integer array should take orders of magnitude longer than a bytearray.

Ideally, as you suggested, we'd have a way to declare an array at a specific address such that it overlays an existing bytearray. I can't think of a way to do this. Something akin to uctypes.bytearray_at but non-allocating, creating a union type construct:

from uctypes import addressof, array_at  # I wish...
a = bytearray(320*240)  # Fast
buffer = array_at('H', addr=addressof(a), 320*120)  # Faster

But given that I don't care about the initial values, array('H', range(320*120)) is significantly faster: about 100 ms.

But it breaks for values > 65535. If you do care about initial values, trivial Viper code can init in ~8ms.

[corranwebster]

Oooo! I didn't know about uctypes... I think that almost gives me what I was wanting:

import uctypes

N = 320*240
s = {'arr': (0 | uctypes.ARRAY, N | uctypes.UINT16)}
b = bytearray(2 * N)
a = uctypes.addressof(b)
c = uctypes.struct(a, s)

Then c.arr is effectively an array of unsigned 16-bit integers sharing the same memory as the bytearray. There's some gorpyness (eg. len(c.arr) doesn't work, you can't iterate over it or slice into it, and memoryview sees it as bytes) and it's incredibly fragile (if b goes away you're looking at what is effectively a random chunk of memory...), but put a wrapper class around it and you can make it more robust and more Pythonic (in particular, you can hold on to the original bytearray and make sure it doesn't get gc'd).

It's good to know that there is a way to do what I wanted, but I will now be responsible and not do that.

I'm not surprised that anything that has to iterate via the Python API is slow: I've spent many years writing CPython code which heavily usedNumPy etc. and very quickly learned that anything that has to use Python looping (even if called from C) will be 100-1000 times slower than the equivalent NumPy code.

In terms of APIs, it's annoying that you can do bytearray(N) to get an array of 0-initialized bytes but not array.array('H', N) to get an array of 0-initialized 16-bit ints, but that's inherited from Python so it would be a moderately hard sell to add it to Micropython unless Python added it first.

All that said, I think that the correct solution architecturally for Tempe is to use raw bytearrays and track the pixel size, since that opens the possibility of supporting other color formats in the future.

Thanks for the insightful discussion!

[corranwebster]

I've just released Tempe 0.3, which includes significant performance improvements over the initial release (up to an order of magnitude in some cases, and twice as fast in most). In particular, text rendering is much faster.

Additional improvements include:

• a proper framework for display drivers, including support for some Pimoroni and Waveshare ST7789 SPI devices (thanks to @JoGeDuBo for information on the Waveshare displays)
• alignment for Text shapes relative to their geometry
• the ability to have Window objects which provide subsurfaces that group objects to facilitate scrolling operations.

Full release notes are on GitHub.