Raspberry Pi 4 build of NixOS appears to be broken (boots to console and then black screen; green LED flashes twice intermittently)

[pmarreck]

Describe the bug

I downloaded build ID 190096830 from https://hydra.nixos.org/job/nixos/trunk-combined/nixos.sd_image.aarch64-linux and burned it to an SD card after updating the rpi4 firmware; it booted the console but at some point after stage 2, the whole screen goes black (there is a single "glitch" on screen before this) and the green LED flashes on-on-pause (red LED stays off) regularly and repeatedly. I tried googling "rpi4 flashes green LED twice continuously with red LED off" and such, but didn't find anything that seemed relevant. Not sure if relevant, but the screen is an UWQHD (3440x1440 native resolution); this screen has worked fine with every other OS I've tried, such as Ubuntu or Manjaro for ARM. I just tried the latest Manjaro ARM build on it and everything works fine, so it does look like an OS build issue/bug.

Steps To Reproduce

Steps to reproduce the behavior:

1. Download the ARM image from above
2. Follow the install instructions at https://nix.dev/tutorials/installing-nixos-on-a-raspberry-pi
3. Hook up a UWQHD display
4. Observe failure to boot to a graphical GUI after Stage 2 of the boot process

Expected behavior

I expected to see a login screen

Screenshots

Blackness

Additional context

The display is an UWQHD (3440x1440) which has worked fine with other RPi4 OS'es (which were probably using the vendor kernel)

Notify maintainers

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

(I can't, because it never booted.)

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
output here

[buckley310]

My pi4 will not output to my 2560x1440 screen when using the official raspberry pi OS lite images. I figured it was just a hardware issue, but I tested today and indeed the official graphical images do work, just the cli-only ones break. There may be some sort of workaround baked into all of these other graphical images for high res screens that we're missing.

I did diff the config.txt files from the two official images, but they are the same.

One thing I did look at was whether or not 32 vs 64 bit makes a difference. It does not seem to. Both of the official cli-only images break while both of the GUI ones work. That's good news.

[pmarreck]

So basically, anything CLI-only that uses the (is it the vendor kernel, or the linux one?) is broken on RPi, at least at these resolutions (2560x1440 or 3440x1440)?

[buckley310]

As far as i can tell, yes.
There are some settings in the config.txt that may help, but have not done extensive testing yet.

Some settings from the default config file that look relevant but are commented by default:

config_hdmi_boost=4

hdmi_safe=1

hdmi_group=1
hdmi_mode=1

I would be surprised if forcing the resolution down didn't make it work, not sure if that matters for people running CLI-only, thus:

framebuffer_width=1280
framebuffer_height=720

[buckley310]

I forgot to ask, are you using a x/wayland desktop, or just booting to TTY?

[pmarreck]

@buckley310
I believe that the nixos build for RPi just boots to TTY and expects you to edit the config file to install/bootstrap whatever else you need, such as your choice of window manager

[tobiasBora]

Some users reported very similar issues in Matrix (boots until Welcome on NixOs message at the end of phase 2 and then the screen disconnects/goes black) and the solution was actually simple: the Raspberry Pi 4 has two HDMI outputs and the boot process seems to output stuff on HDMI 0 but after the boot the HDMI 1 is used instead for the TTY prompt… I'm not sure if you also have the same issue, let us know.

If possible it may be interesting to output to the HDMI that is actually plugged. If it's not possible, maybe write a warning message, or at least redirect to HDMI 0 (that sounds more logical).

This bug may actually be related to #179701

[pmarreck]

Interesting. I will try switching HDMI outputs when things go black.

[i-am-logger]

As far as i can tell, yes. There are some settings in the config.txt that may help, but have not done extensive testing yet.

Some settings from the default config file that look relevant but are commented by default:

config_hdmi_boost=4

hdmi_safe=1

hdmi_group=1
hdmi_mode=1

I would be surprised if forcing the resolution down didn't make it work, not sure if that matters for people running CLI-only, thus:

framebuffer_width=1280
framebuffer_height=720

tried these though still experiencing screen goes dark and monitor goes to sleep.

I can confirm this did not help with a 4k display or wide screen display.

Just FYI
Original imaged (with suggested changes to config.txt) worked fine with a 7" 1280x720 i have in hand

[tobiasBora]

@ibootstrapper have you tried to use the second HDMI output as recommended above?

[RyanGibb]

@ibootstrapper @tobiasBora I've just checked and using HDMI-1 as a workaround doesn't work on a 4k monitor for me either

[i-am-logger]

above?

Yes, both HDMI ports are taking the monitor for a nap :-)

[i-am-logger]

I only had to change the password to the nixos user and then I could ssh to the pi via Ethernet . So I'm wondering maybe we can put a default password for the image to allow ssh to work without HDMI needs or TTY? This anyhow will change once applying of the real configuration . Just a thought ...

[RyanGibb]

@ibootstrapper if I understand correctly you're taking about building a custom sd-image with Nix? This is indeed possible, see here for an example. You can build it with nix build '/etc/nixos#nixosConfigurations.iso.config.system.build.isoImage. You can even configure a WiFI password for the image so you can SSH in wirelessly without the need to connect a display.

[i-am-logger]

Perfect, thank you

[WizardUli]

I have this same problem with 1400x1050 display. It is not exclusive to high resolutions then.

[Tungsten842]

I am getting exactly the same issue on a Full HD display.

[pmarreck]

It's entirely NixOS's fault... that I want to use it on every one of my computers, even my raspi's. 😀

[Tungsten842]

Anyway, I was able to boot nixos using the official uefi image with https://github.com/pftf/RPi4, so the bug is specific to the sd image somehow.

[WizardUli]

boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4; also works

[RyanGibb]

@WizardUli when you say works, you mean it solves this problem?

[WizardUli]

@WizardUli when you say works, you mean it solves this problem?

Well, if the problem is the black screen, then yes. It works for me on Pi4 on 1400x1050, on FHD, and on 1440p. The resolution is not correct though (no modesetting) but readable. If you want a correct resolution then you need to add kms dt overlay and currently both the default nixpkgs' algorithm for applying device tree overlays is broken as well as nixos-hardware's one NixOS/nixos-hardware#503 and you need to rewrite it (the nixos-hardware's one) slightly so it does not try to write to an output of another derivation 😂

[tobiasBora]

@WizardUli can you maybe put here detailled instructions how you solve the issue? I'm sure it would be extremely helpful to many people.

[RyanGibb]

I can confirm adding boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4; to my nixos config solves this problem for me.

[matthew-salerno]

It seems UART isn't enabled in the kernel either, I thought I was being clever. I get no output after uboot hands control over to the kernel. This might be a separate issue.
Building the iso doesn't appear to render a pi compatible image with a FIRMWARE partition. I'm not very familiar with nixos at this level so maybe I'm doing something wrong. I take it the best way forward for new installs is to use the older working image, add boot.kernelPackages = pkgs.linuxKernel.packages.linux_rpi4;, and update from there?