low-level screen improvements

[tlubke]

Gamma

Related PRs:

• Screen gamma + "DISPLAY" setting menu. #1583
• Modify ssd1322 driver to use gamma + added precharge voltage device file. linux#47

TODO

• driver changes
  • support gamma in device tree
  • support precharge voltage in device tree
  • check if user space can write to "gamma" and "precharge" device tree files if suggested changes are made in norns to use write() and close() Screen gamma + "DISPLAY" setting menu. #1583 (comment)
• norns core
  • c functions for setting gamma (done, but changes requested)
  • lua-c binding
  • "DISPLAY" system settings menu
    • UI
    • persistence
    • SYSTEM > RESET resets display settings Screen gamma + "DISPLAY" setting menu. #1583 (review)
  • limit "precharge" to useable range (0-31 -> 16-31, seen by user as 0-15) and refer to as "brightness".
• userspace "gamma" + "precharge" disables itself on fates Modify ssd1322 driver to use gamma + added precharge voltage device file. linux#47 (comment)
• investigate wide text occurring with A4 and A8 drawing surfaces.

Cairo Surface

Related PRs

• https://github.com/freedesktop/cairo/pull/3
• SD1322 driver changes to accomodate cairo A4. linux#48
• Matron changes to support cairo A4. #1601

TODO

• add support for A4 to cairo
• compile updated cairo on norns
• link newly compiled cairo lib on norns
• modify norns core to use A4 surfaces
• modify screen driver to use framebuffers compatible with A4 cairo surface
• fix libcairo A4 branch, clearing the surface doesn't work correctly

Frame Rate

TODO

• modify screen driver to operate at 60hz
• implement an optimized framebuffer algorithm for power/cpu efficiency

Update Procedure

possibly a separate issue, see: monome/linux#47 (comment)

[tlubke]

made a lot of progress on the cairo A4 format item this week. the norns side is pretty much implemented, just have to go back to my libcairo branch now and find what I missed.

[tlubke]

Found there was nothing wrong with my cairo branch in the end, it just needs to be configured with XCB surfaces.

sudo apt-get install libxcb1-dev libxcb-render0-dev
cd cairo; sudo ./autogen.sh --enable-xcb

However, I found with A8 and A4 drawing surfaces, some screen text appears to be double width. I think it may have something to do with anti-aliasing. I'm looking into it now.

[tlubke]

However, I found with A8 and A4 drawing surfaces, some screen text appears to be double width. I think it may have something to do with anti-aliasing. I'm looking into it now.

This also appears to be a compilation issue, using the standard package with the same norns code doesn't have this result. Essentially when the text extents of screen.text_center() aren't integer numbers, the text bleeds into other pixels.

I'll see if I can find what options the apt package is compiled with.

[tlubke]

Successfully tested 60 FPS this morning. Only change to the driver was actually clocking down the refresh rate a bit ~111Hz to ~61 Hz. A change to the SPI overlay is what traverses through FBTFT. No noticeable increase in battery usage.

FullSizeRender.MOV

Will put together a fresh image with gamma controls, Cairo changes, and the FPS all rolled in. After that gonna work on compiling the driver out of tree, and see if I can make it installable with a Debian package. If not, I may try building my own SPI driver.

This is captured with my phone at 120 fps. Just a simple script that increments a number every frame at 60 fps.

[tlubke]

https://github.com/tlubke/norns/releases/tag/220802

[tehn]

very cool, and surprised the driver didn't need a more major change

i can test out the image here. the video above is a bit confusing, but perhaps it's a web codec issue on my end? it counts up to 60 twice over the course of 10 seconds?

[tlubke]

... perhaps it's a web codec issue on my end? it counts up to 60 twice over the course of 10 seconds?

@tehn
Sorry, should've left more of an explanation. This is just proof via slow-motion capture that every frame counting up to 60 is able to be shown, I just let it loop twice. Running the same script without the image shows only every 3rd or 4th number.

Script:

c = 0
fps = 60

function init()
  clock.run(function()
    while true do
      clock.sleep(1/fps)
      redraw()
      c = c+1
    end
  end)
end

function redraw()
  screen.clear()
  screen.move(32,32)
  screen.text_center(c % fps)
  screen.update()
end

[tehn]

apologies for the delay. just checking the build info for the image you posted: is it for a shield?

copied it to a standard norns and getting this error when trying to run matron (from the command line, since the system service failed):

build/matron/matron: error while loading shared libraries: /lib/arm-linux-gnueabihf/libpixman-1.so.0: invalid ELF header

[tlubke]

@tehn

apologies for the delay. just checking the build info for the image you posted: is it for a shield?

No, the image should be for the standard norns. I will double check the image later today, you can reinstalling libpixman-1-dev with apt to see if that fixes the error.

I've also thought of another way of accomplishing this that perhaps I could get your thoughts on, since it wouldn't require changes to the kernel.

We can change the device overlay (it's not built into the kernel) to make the SPI behave as just a generic SPI device, and interact with the screen in the userspace via matron. This not only opens up a lot more possibilities for future improvements, but also removes the need for a frame buffer abstraction in the first place since we could copy the cairo surface directly from screen.update(). Additionally the gamma and brightness settings would become much more straightforward, and one would be able to implement things like adaptive refresh rates or frame limiting.

I was working on this a bit last week. I can share more when I have a proof of concept.

[tehn]

@tlubke thanks for the tip, i'll attempt to re-install the lib

honestly i'd very much like to avoid a kernel change if possible--- we just recently put the entire userbase through re-imaging and it's perceived (rightly so) as a hassle. but i think 60fps is worth it, personally :)

so yes, i'm very interested in your proposal. i don't have any issues philosophically with userspace SPI. in general i'd like to think of the norns more as an embedded system rather than a computer (if that makes sense).

[tlubke]

@tehn I know norns hardware isn't open source, but is there a schematic I could reference? Is it the same pin layout as the shield?

I've got SPI devices set up in /dev/, and I'm controlling the GPIOs for DC and Reset. Not sure, but I think I might need to know chip select pins.

[tehn]

here's the pinout: https://github.com/monome/norns-image/blob/main/readme-hardware.md#pinout-1 let me know if you need more, happy to provide

…

On Tue, Oct 18, 2022 at 6:27 PM tlubke ***@***.***> wrote: @tehn <https://github.com/tehn> I know norns hardware isn't open source, but is there a schematic I could reference? Is it the same pin layout as the shield? — Reply to this email directly, view it on GitHub <#1600 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB4I4EB2HQ5XEJBTL3SLW3WD4P4JANCNFSM6AAAAAAQKYXZLM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tehn]

wow. drawing some 60 fps circles with slowly pulsing radii. it's really, really impressive!

it'd be great to have some broader testing with this big change--- i'm curious to see how certain heavy scripts react to the new capability. ie, if people can easily convert their scripts to 60fps with a few character edits, this might have ripple effects on the performance of their script. so we may need to do some more docs on optimization.

(ps, ended up my first download of the image was corrupted, hence my troubles)

any further insights into userland spi?

[tlubke]

@tehn

it'd be great to have some broader testing with this big change--- i'm curious to see how certain heavy scripts react to the new capability. ie, if people can easily convert their scripts to 60fps with a few character edits, this might have ripple effects on the performance of their script. so we may need to do some more docs on optimization.

Agreed. I think optimization docs should come first, but having SPI in the userspace might allow matron to smartly throttle things and give users a warning. The few scripts I've seen that come closing to maxing out the CPU don't have very intense screen operations. I assume there is no current priority to one system over another (i.e. screen over clock over super collider)?

any further insights into userland spi?

I believe I'm close, but I may need some help or some more eyes to see what I'm missing. I've tried both using linux IOCTL, and using the BCM2835 library. I just uploaded branches. As a proof of concept, I am trying to turn on "ALL ON" mode, and get the whole screen to light up, the branches only set normal mode like in the driver init.

For both of the branches, I experimented with setting different SPI modes, chip selects, and write speeds.

IOCTL

Default SPI enabled via raspi-config and editing /boot/config.txt

...
# Display
# Connected using SPI
dtparam=spi=on
#dtoverlay=ssd1322-spi

https://github.com/tlubke/norns/tree/tlubke/userspace-spi-IOCTL

I am able to see that the right amount of bytes are being written to spidev0.0, and I don't see any error messages on the GPIO writes. Nothing appears to happen on the screen though.

BCM2835

Since I saw we use the driver in the overlay, figured I would try using the library. Use of the library's SPI function requires that SPI be disabled in raspi-config, and matron has to be run as sudo. I didn't have access to a device file system to check if bytes were being written, so it was a bit of a shot in the dark.

https://github.com/tlubke/norns/tree/tlubke/userspace-spi-bcm2835

Side note: I see there is a discussions page in the norns repo now, should I move this issue there (if that is easy/possible)?

[tehn]

this would be a great use of the discussions page!

…

On Tue, Oct 25, 2022 at 2:56 PM tlubke ***@***.***> wrote: @tehn <https://github.com/tehn> it'd be great to have some broader testing with this big change--- i'm curious to see how certain heavy scripts react to the new capability. ie, if people can easily convert their scripts to 60fps with a few character edits, this might have ripple effects on the performance of their script. so we may need to do some more docs on optimization. Agreed. I think optimization docs should come first, but having SPI in the userspace might allow matron to smartly throttle things and give users a warning. The few scripts I've seen that come closing to maxing out the CPU don't have very intense screen operations. I assume there is no current priority to one system over another (i.e. screen over clock over super collider)? any further insights into userland spi? I believe I'm close, but I may need some help or some more eyes to see what I'm missing. I've tried both using linux IOCTL, and using the BCM2835 library. I just uploaded branches. As a proof of concept, I am trying to turn on "ALL ON" mode, and get the whole screen to light up, the branches only set normal mode like in the driver init. For both of the branches, I experimented with setting different SPI modes, chip selects, and write speeds. IOCTL Default SPI enabled via raspi-config and editing /boot/config.txt ... # Display # Connected using SPI dtparam=spi=on #dtoverlay=ssd1322-spi https://github.com/tlubke/norns/tree/tlubke/userspace-spi-IOCTL I am able to see that the right amount of bytes are being written to spidev0.0, and I don't see any error messages on the GPIO writes. Nothing appears to happen on the screen though. BCM2835 Since I saw we use the driver in the overlay, figured I would try using the library. Use of the library's SPI function requires that SPI be disabled in raspi-config, and matron has to be run as sudo. I didn't have access to a device file system to check if bytes were being written, so it was a bit of a shot in the dark. https://github.com/tlubke/norns/tree/tlubke/userspace-spi-bcm2835 Side note: I see there is a discussions page in the norns repo now, should I move this issue there (if that is easy/possible)? — Reply to this email directly, view it on GitHub <#1600 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB4I4CJ4Q2ONVXN6BHAOVTWFAUMNANCNFSM6AAAAAAQKYXZLM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tlubke]

@tehn @dndrks

I don't think I have the repo permissions to do this. Could one of you convert the issue when you find the time?

https://docs.github.com/en/discussions/managing-discussions-for-your-community/moderating-discussions#converting-an-issue-to-a-discussion

[tlubke]

I wonder if it's an endian issue, isn't rpi little endian? Setting SPI_IOC_WR_LSB_FIRST isn't working for me, however.

https://www.kernel.org/doc/Documentation/spi/spidev

[tlubke]

Now trying this in /boot/config.txt based on some googling, but unable to get spi_bcm2835 to show up in lsmod.

dtparam=spi=on
dtoverlay=spi-bcm2835-overlay

EDIT: nevermind, spi-bcm2835 is built into the kernel and loaded by default. Supposedly won't show up in lsmod.

[tlubke]

Hooked up my norns to an oscilloscope to compare readouts of the SPI pins with the driver vs userspace. Got the tlubke/userspace-spi-bcm2835 branch looking pretty close to what I saw with the driver, the ioctl branch didn't seem to pull CS0 low on write, not sure why.

Still don't quite know what is wrong, DB0 (SCLK) and DB1 (SDIN) look odd... but SCLK didn't look like how I expected it to when scoping the driver.

I can't tell if the screen is supposed to be using an internal clock or an external one (mentioned in the datasheets). If it's external, perhaps clock settings need to be more exact?

[tlubke]

New SPI Driver?

Tabling work on userspace SPI right now since I haven't made any progress in a few days. Attempting to write a kernel SPI protocol driver from scratch. Could hopefully then compile it outside the kernel, add it to the existing modules, and change out the overlay.

The SPI and framebuffer (maybe I'll do Kernel Mode Setting instead, linux people seem to be adamant fb shouldn't be used anymore) parts seem pretty straightfoward, I don't know much about device tree / overlay yet though. If anyone wants to learn this along with me, just reach out.

As far as device files go, I was thinking:

• fps
• refresh-rate (hz)
• gamma
• contrast
• precharge-voltage

But maybe I should make a device file for all the possible settings mentioned in the SSD1322 datasheet to leave it open to future development?

Additional Findings on 60FPS Image

User static#2765 on the norns discord found some issues.

I’ve noticed some graphical glitches in testing with a few scripts so far though - such as the trailing of animated elements, or a fully white display. Also, sometimes flickering and sometimes blurry lines (but the blur may be expected in some instances)
an example of trailing is with p8/ghosts. Icarus is the one I noticed flickering with. Massif has a fully white display. Initenere has some blurry / drop-shadow looking thing happening.

I'm fairly certain the anti-aliasing on A8 is wrong, which might explain the trailing animated element part. I have a hunch the fully white display and other white blocks (image below) might be from the .png import function, which I didn't test. Cairo might not be able to implicitly convert ARGB .pngs to just an alpha channel. Should be able to fix the conversion in screen.c if this is the case though.

[tehn]

apologies--- i haven't had a lot of time to do low-level investigation of this (which requires a certain deep focus)

i'm not sure what you meant by external vs. internal clock? the screen is an SPI device in follower mode, so the CM3 will provide the clock.

[tehn]

don't have the hardware in front of me, but is there an spi group?

ie

sudo adduser we spi

[tehn]

oof also found this: https://forum.up-community.org/discussion/2141/solved-tutorial-gpio-i2c-spi-access-without-root-permissions

[tlubke]

Thanks that set me in the right direction, the bcm2835_init() function wants access to /dev/mem and /dev/gpiomem which are the kmem and gpio groups respectively. There is an explicit check in the source code for UID = 0 (i.e. root), but I may be able to add we to those groups, and modify the source code and have it work.

I'm wondering if use of this lib won't conflict with the other peripherals though, since i2c uses the bcm2835 driver, and the keys/encoders use GPIO.

[tlubke]

I may be able to add we to those groups, and modify the source code and have it work.

Hmm, no that did not work. Looks like a kernel recompile would be needed to make this work because of CONFIG_STRICT_DEVMEM.

Looks like I will need to hunker down and figure out why spidev + IOCTL isn't working the same way. I have a feeling it is chip-select related.

[tlubke]

Just had success getting screen to work in userspace using IOCTL! The problem was the GPIOs for CS0 and D/C weren't set active low. I was able to set it through raspi-gpio, maybe better to do this through an overlay somehow in `/boot/config.txt, will figure out later.

commit: tlubke@06c1495

/boot/config.txt

dtparam=spi=on
dtoverlay=spi0-1cs,cs0_pin=7 # Don't use GPIO8, so it stays active low.

[tlubke]

userspace SPI working with matron: tlubke@2581a25

Should work on any norns, will just require sudo apt install libgpiod to compile. Remember to rerun ./waf configure in the norns directory.

Will be adding/modifying the gamma+brightness code and so on after this, then we can start the discussion on what code relating to fbdev can be cleaned up. I know there is at least one mod dependent on /dev/fb0 (isn't created without the fbtft driver), but I think norns desktop works without it it?

There is also not currently a frame rate limit in place, and I haven't dropped the refresh rate in the same I did on the 60FPS image.

If anyone has any insight on why setting spidev.bufsize=8192 in /boot/cmdline.txt isn't working, that would be helpful as well.

[tehn]

amazing! i've never had to alter the buffer size for spidev, so i don't much much insight there at this point.

which mod is dependent on fb0?

[tlubke]

which mod is dependent on fb0?

I thought I remembered ndi-mod hooking into the framebuffer, but I just checked the lines thread and it is actually getting the cairo surface directly. There aren't any others that I know of.

I noticed there are some C++ classes in crone but none in matron. Need to make the functions in ssd1322.c thread safe, and was wondering if I could use C++ in matron at all or if it should be kept strictly C? Would like to use std::mutex namely.

EDIT: nevermind on C++, got it figured out with pthread_mutex_t.

https://github.com/tlubke/norns/tree/tlubke/wip-screen-improvements will be my working branch until I get ready for a PR.

[tehn]

checked out your branch, the code looks good. it feels good to have the screen exposed at a lower level for this project.

[tlubke]

checked out your branch, the code looks good. it feels good to have the screen exposed at a lower level for this project.

Thank you, I am grateful for the feedback. And I agree, I think it makes the screen much more accessible having it out of the kernel space. Took me months to figure out all that stuff (though I learned a lot in the process). I was worried transfer speed might suffer in user space, but I have yet to see any hiccups.

I have made a few more commits since you checked out the branch, and there is still a bit of work to do. Notably, I had to change the default cairo operator to source. Overall, I think it's at the point where it can be beta tested.

TODO:

• test other cairo/PNG formats with screen.display_png()
• query existing scripts for use of screen.blend_mode("over") since it doesn't work the same way with alpha-only format
• (?) add weaver binding for ssd1322_set_contrast()
• add frame-rate limit to screen_update()
• add adaptive refresh-rate algorithm to optimize battery life
• rerun luadoc on the branch to pickup screen lua doc changes
• (?) add more annotations to definitions in ssd1322.h for easier reference
• (?) raise minimum of gamma setting to 1.0 (default, linear)
• (!) fix norns shield orientation (upside-down for @eigen).
  There is probably more that could be done, but this is all I can think of at the moment.

[tlubke]

copied from discord

How to test experimental screen improvements:

1. SSH into your norns
2. edit /boot/config.txt and put an # in front of the line dtoverlay=ssd1322-spi.
3. edit /boot/cmdline.txt and add spidev.bufsiz=8192 to the end of the file.
4. cd ~/norns
5. git remote add screen-test https://github.com/tlubke/norns
6. git fetch screen-test
7. git pull
8. git checkout tlubke/wip-screen-improvements
9. sudo apt install libgpiod-dev
10. ./waf configure
11. ./waf
12. restart your norns

How to revert back to stable norns:

1. SSH into your norns
2. edit /boot/config.txt removing the # from the line #dtoverlay=ssd1322-spi.
3. cd ~/norns
4. git checkout main
5. ./waf
6. restart your norns

[tlubke]

Was able to expand the spidev buffer size by adding this to /boot/cmdline.txt:

spidev.bufsiz=8192

Not sure why it wasn't working for me before, maybe I was spelling it bufsize instead bufsiz.

Added a frame limiter to screen_update() over the weekend, initially thought it couldn't keep up at 60fps. But it turns out that using clock with sleep(1/60) to redraw isn't as accurate as using a metro callback. Might be worth investigating if that isn't already a known issue.

[Dewb]

This looks great @tlubke! You poked me at the exact right moment, I finally had some time today to try it out. I have been lurking this conversation (on github and on Discord) and thinking about the implications for ndi-mod but haven't had much to say until I had a chance to get hands on with it.

In general you should definitely do what's best for the base case and not worry about ndi-mod, it can probably adapt to whatever.

One thing that made ndi-mod efficient was that I could hand the buffer pointer from the Cairo surface directly to the NDI library, since the ARGB buffer can be interpreted as RGBX (when A=R=G=B.) There is no direct equivalent of A8 in the set of NDI-supported frame formats, but there are a couple that are close. NV12 has a Y8 luma plane followed by a subsampled UV plane. So we can interpret the Cairo buffer as NV12 if we construct an appropriate chroma plane that sits directly after it in memory.

Here's an implementation of that approach; this branch allows ndi-mod to work with the wip-screen-improvements branch.
https://github.com/Dewb/ndi-mod/tree/norns-60fps

The cost of this is an 8KiB memcpy on every frame, and 12KiB of allocated buffer (plus additional costs for extra Cairo surfaces that are added to NDI via script.) This copy doesn't seem to have a visible effect on the CPU meters.

Another strategy would be for norns to allocate the extra buffer space to allow each Cairo surface to be interpretable as an NV12 frame. This would mean only an additional 50% of memory for each surface (instead of 150%) and no memcpys at all. Here's a hacky implementation of that approach that also works: https://gist.github.com/Dewb/6c33c28d798eaa5d50ee421e02315a45

A more plausible proposal would be some kind of hook to allow a mod to substitute its own backing buffer creation routine, but I'm not sure it's worth the effort.

On switching to A4: this would be a little more expensive for ndi-mod, as NDI doesn't support any 4bpp formats so we'd have to get into bit-shifting to construct an 8bpp frame. I notice that the Cairo A4 changes involve a lot of shifting on write as well, it's not clear to me that that would overall be a performance win.

Finally, this isn't specific to the 60fps changes, but a related enhancement I'd like to propose is a mod hook for screen updates that is rate-limited just like the screen, but is not disabled by the screensaver. This would serve two purposes: One, multiple mods could use the hook instead of replacing screen.update, so they would no longer interfere with each other. Two, the same frames would be sent to the screen and the NDI stream. Right now, with separate rate-limiting, NDI and the screen could decide to drop different frames. Adding this hook would be pretty straightforward except for the "is not disabled by the screensaver" part, because the screensaver is implemented in lua and replaces screen.update. The screensaver logic would probably have to move to the C side.

[tlubke]

Took a bit of a break from this because of the holidays, but I have a small update.

@Dewb thanks for your thoughts! I have a couple things to address:

One thing that made ndi-mod efficient was that I could hand the buffer pointer from the Cairo surface directly to the NDI library, since the ARGB buffer can be interpreted as RGBX (when A=R=G=B.) There is no direct equivalent of A8 in the set of NDI-supported frame formats, but there are a couple that are close. NV12 has a Y8 luma plane followed by a subsampled UV plane. So we can interpret the Cairo buffer as NV12 if we construct an appropriate chroma plane that sits directly after it in memory.

Both good/bad news, I unfortunately hadn't tested CAIRO_FORMAT_A8 with the blending modes until a day or two after this post -- I found that they don't work with them at all, so I'm reverting the cairo formats back to ARGB32. This means that you wouldn't need to make the change to NV12 to keep ndi-mod compatible. However, since I have to convert the ARGB into an 8KiB buffer anywhere for transfer via SPI, we could provide a way to pass that pointer instead of the cairo surface. This could help CPU cost down if it is an issue at 60fps. I made a simple function, ssd1322_resize_buffer(size_t) that will reallocate, reassign, and return the pointer. That way, you could avoid the memcpy()s and the extra memory won't be used until needed. Similar to what you suggested a bit further down about mod hooks.

Finally, this isn't specific to the 60fps changes, but a related enhancement I'd like to propose is a mod hook for screen updates that is rate-limited just like the screen, but is not disabled by the screensaver. This would serve two purposes: One, multiple mods could use the hook instead of replacing screen.update.

This could potentially be simply fixed by turning off the screen (would have to make another function in ssd1322.h probably) instead of changing the screen.update assignment to a different function when in screensaver mode. Could even cut off the SPI traffic.

---

Also want to thank @Dewb for helping me find some ARM neon instructions to make RGB conversion efficient.

[tlubke]

Opened #1650.

Still needs to be tested with desktop norns if someone can help me with that.