Balena.io layers for Yocto

Description

This repository enables building balenaOS for various devices.

Layers Structure

• meta-balena-common : layer which contains common recipes for all our supported platforms.
• meta-balena-* : layers which contain recipes specific to yocto versions.
• other files : README, COPYING, etc.

Dependencies

• https://docs.yoctoproject.org/brief-yoctoprojectqs/#build-host-packages
• docker
• jq

Versioning

The meta-balena version is kept in the DISTRO_VERSION variable. The balena-<board> version is kept in the file called VERSION located in the root of the balena-<board> repository and read in the build as the variable HOSTOS_VERSION.

• The version of meta-balena is in semver format being 3 numbers separated by a dot. The patch number can have a beta label. e.g. 1.2.3, 1.2.3-beta1, 2.0.0-beta1.
• The first balena-<board> release based on a specific meta-balena release X.Y.Z, will be X.Y.Z, the same as the meta-balena version. Example: the first balena-<board> version based on meta-balena 1.2.3 will be 1.2.3.
• Subsequent balena-<board> releases are constructed by appending to the meta-balena version a rev label. For example a meta-balena 1.2.3 can go through 3 board revisions, being 1.2.3 the initial revision, and 1.2.3+revN the subsequent ones, with the final version being 1.2.3+rev2 .
• When updating meta-balena version in balena-<board>, the version will reset to the meta-balena version. Ex: 1.2.3+rev4 will be updated to 1.2.4 .

We define host OS version as the balena-<board> version and we use this version as HOSTOS_VERSION.

Build flags

Before bitbake-ing with meta-balena support, a few flags can be changed in the conf/local.conf from the build directory. Editing of local.conf is to be done after source-ing. See below for explanation on such build flags.

Configure custom network manager

By default balena uses NetworkManager on host OS to provide connectivity. If you want to change and use other providers, list your packages using NETWORK_MANAGER_PACKAGES. You can add this variable to local.conf. Here is an example:

NETWORK_MANAGER_PACKAGES = "mynetworkmanager mynetworkmanager-client"

Customizing splash

We configure all of our initial images to produce a balena logo at boot, shutdown or reboot. But we encourage any user to go and replace that logo with their own. All you have to do is replace the splash/balena-logo.png file that you will find in the first partition of our images (boot partition) with your own image. NOTE: As it currently stands plymouth expects the image to be named balena-logo.png. In older releases this file was called resin-logo.png.

Docker storage driver

By default the build system will set all the bits needed for the docker to be able to use the aufs storage driver. This can be changed by defining BALENA_STORAGE in your local.conf. It supports aufs and overlay2.

OS development

To configure a development build that disables quiet boot and allows bootloader shell access, edit the build's local.conf adding:

OS_DEVELOPMENT = "1"

This is a development only setting and no OS_DEVELOPMENT configured images are deployed.

The OS

SSH and Avahi services

The OS runs SSH (openSSH) on port 22222. Running this service takes advantage of the socket activation systemd feature so the SSH daemon will only run when there is a SSH connection to the device saving idle resources in this way. In order to connect to a device, one can use it's IP when known or resolve the hostname over mDNS as its hostname is advertised over network using an avahi service. When the latter is used, configuration of the client is needed (see for example https://wiki.archlinux.org/index.php/Avahi#Hostname_resolution).

Time synchronization

At boot, time is set and synchronized as follows:

• Build time
• Previous boot system time
• RTC time when available
• HTTPs time
• Network time

Build time

Initially time is set from the image build time timestamp, stored in /etc/timestamp and generated by the build system when the image is generated.

Previous boot system time

The system then checks whether the previous boot system time was stored in persistent storage and uses it to correct the time. Time is stored in persistent storage on an hourly timer and on system reboot or shutdown. Logging starts after the last boot system time is set.

RTC time

When an RTC is available (/dev/rtc), a timeinit-rtc service is started that updates the system clock using the value read from the RTC. If there is no RTC available, the service will not do anything.

HTTPs time

After a network connectivity event, an HTTPs time synchronization service timesync-https, is then used to correct the time from HTTP headers timestamps, assuming the correct system time has not been set from the RTC previously. This guarantees the time is broadly correct and certificates expiration checks won't fail. Other network services are held until this time synchronization happens. By default, the time synchronization uses the NetworkManager connectivity URL defined in the connectivity section of config.json. To disable the HTTPs time sync and allow other services to run, set the connectivity check URI to 'null'. This will also disable connectivity checks too.

Network time

The chronyd services is responsible of managing the time afterwards using NTP. It is configured to synchronize every 4h approximately to save bandwidth. If the NTP servers become unreachable, the service will continuously try to update the time. If the time gets unsynchronized, the NTP client service will be restarted to correct failures.

The time keeping framework explained above provides robust time initialization and management both when RTC is available and when not.

Bootloader

The bootloader needs to select the active root filesystem, load, and launch the Linux kernel. It also manages boot counts and rollbacks. BalenaOS supports several bootloaders across the supported devices line-up.

• Balena bootloader

  • This is the preferred bootloader for new kexec capable devices
  • It is a minimal Linux kernel that contains all balena's logic, like rollback support, in the initramfs so that it does not have to be replicated across different bootloaders
  • It fetches the final kernel from the active root partition and kexecs into it
  • The balena bootloader is a requirement for secure boot enabled platforms as it needs to mount and decrypt root filesystems to launch the final kernel. Decrypting disks is not usually supported on standard bootloaders.
  • It also provides a central environment file, bootenv, to perform bootloader configuration by the system
  • Ideally, the balena booloader can be built as an EFI binary and directly launched by an EFI capable bootROM
  • Alternatively, a vendor bootloader like U-Boot can be minimally configured to launch it
  • Using the balena bootloader reduces the friction of porting new hardware as modifications to vendor bootloaders are minimal, usually just launch configuration, and all balena logic has already been implemented and tested, and is the same across all devices

• Other bootloaders supported that can be used to launch the balena bootloader if needed, or on non-kexec capable devices are:

• U-boot

  • Is used on most of the legacy supported ARM device-types
  • Only block devices can be used with BalenaOS, RAW flash devices are not supported
  • Common functionality is implemented in the u-boot environment, which is provided by the common OS Yocto layer
  • The environment is embedded in the u-boot binary. This allows for the intended configuration to be used with the matching version of BalenaOS and avoids interference from any pre-programmed environment
  • Device-specific functions are provided by the device repository, either in a u-boot script or in the enviroment defined by the board configuration files
  • Specific Jetson modules (TX2, Nano) use an extra extlinux.conf file, which is loaded and parsed by u-boot
  • Three environment files are stored and loaded by u-boot from the BalenaOS boot partition. resinOS_uEnv.txt is used for storing the active root partition index, extra_uEnv.txt stores device-specific configuration elements like optional kernel command-line parameters as well as any custom selected device-tree while bootcount.env stores the number of failed attempted boot retries during an OS update. NOTE: Custom device-tree selection is supported only on specific devices
  • Applies the kernel device-tree overlays specified in uEnv.txt_internal/uEnv.txt on Beaglebone devices

• Grub

  • Is used for the legacy supported x86 device-types
  • Common functionality is implemented by the OS layer in the grub configuration template

• Cboot

  • Is used on Jetson Xavier devices running L4T 32.X
  • Loads device-trees from device-specific A/B partitions
  • Unlike the rest of the bootloaders, it does not support FAT filesystems
  • The current active root filesystem label is defined in the kernel command line provided by the kernel device-tree. The active rootfs is selected at boot time based on the active device-tree

• UEFI L4Tlauncher

  • Is used on the Jetson Orin platforms
  • Obtains the kernel image path and kernel cmdline arguments from extlinux.conf files
  • Rollbacks and active root filesystem selection are implemented in bootloader patches provided bythe balena-jetson-orin device repository
  • Employs the same rollback mechanisms used by balenaOS in u-boot by storing and reading environment variables from the resinOS_uEnv.txt, extra_uEnv.txt and bootcount.env files

Rollback framework

Check docs/rollbacks.md for the rollback documentation

Devices support

WiFi Adapters

We currently tested and provide explicit support for the following WiFi adapters:

• bcm43143 based adapters

Modems

We currently test as part of our release process and provide explicit support for the following modems:

• USB modems (tested on Raspberry Pi 3, Balena Fin, Intel NUC and Nvidia TX2)
  • Huawei MS2131i-8
  • Huawei MS2372
• mPCI modems (tested on Balena Fin and Nvidia TX2 Spacely carrier)
  • Huawei ME909s-120
  • Quectel EC20
  • SIM7600E

Recommended WiFi USB dongle

• Panda N600 Dual-Band (2.4 GHz + 5 GHz) Wireless-N USB Adapter This USB dongle is based on the Ralink RT5572 chipset and is supported by the generic rt2800usb driver. Tests have been done on the PAU09 model of the Panda N600 Dual-Band USB Adapter and having the firmware version 0.36 from firmware file rt2870.bin

How to fix various build errors

• Supervisor fails with a log similar to:

Step 3 : RUN chmod 700 /entry.sh
---> Running in 445fe69866f9
operation not supported

This is probably because of a docker bug where, if you update kernel and don't reboot, docker gets confused. The fix is to reboot your system. More info: http://stackoverflow.com/questions/29546388/getting-an-operation-not-supported-error-when-trying-to-run-something-while-bu

config.json

The behavior of balenaOS can be configured by setting the following keys in the config.json file in the boot partition. This configuration file is also used by the supervisor.

hostname

(string) The configured hostname of the device, otherwise the device UUID is used.

persistentLogging

(boolean) Enable or disable persistent logging on the device - defaults to false. Once persistent journals are enabled, they end up stored as part of the data partition on the device (either on SD card, eMMC, harddisk, etc.). This is located on-device at /var/log/journal/<uuid> where the UUID is variable.

country

(string) Two-letter country code for the country in which the device is operating. This is used for setting the WiFi regulatory domain, and you should check the WiFi device driver for a list of supported country codes.

ntpServers

(string) A space-separated list of NTP servers to use for time synchronization. Defaults to resinio.pool.ntp.org servers:

• 0.resinio.pool.ntp.org
• 1.resinio.pool.ntp.org
• 2.resinio.pool.ntp.org
• 3.resinio.pool.ntp.org

dnsServers

(string) A space-separated list of preferred DNS servers to use for name resolution.

• When dnsServers is not defined, or empty, Google's DNS server (8.8.8.8) is added to the list of DNS servers obtained via DHCP or statically configured in a NetworkManager connection profile.
• When dnsServers is "null" (a string), Google's DNS server (8.8.8.8) will NOT be added as described above.
• When dnsServers is defined and not "null", the listed servers will be added to the list of servers obtained via DHCP or statically configured via a NetworkManager connection profile.

balenaRootCA

(string) A base64-encoded PEM CA certificate that will be installed into the root trust store. This makes the device trust TLS/SSL certificates from this authority. This is useful when the device is running behind a re-encrypting network device, like a transparent proxy or some deep packet inspection devices.

"balenaRootCA": "4oCU4oCTQkVHSU4gQ0VSVElGSUNBVEXigJTi..."

developmentMode

To enable development mode at runtime:

"developmentMode": true

By default development mode enables unauthenticated SSH logins unless custom SSH keys are present, in which case SSH key access is enforced.

Also, development mode provides serial console passwordless login as well as an exposed balena engine socket to use in local mode development.

os

An object containing settings that customize the host OS at runtime.

network

wifi

An object that defines the configuration related to Wi-Fi.

• "randomMacAddressScan" (boolean) Configures MAC address randomization of a Wi-Fi device during scanning

The following example disables MAC address randomization of Wi-Fi device during scanning:

"os": {
 "network" : {
  "wifi": {
    "randomMacAddressScan": false
  }
 }
}

connectivity

An object that defines configuration related to networking connectivity checks. This feature builds on NetworkManager's connectivity check, which is further documented in the connectivity section here.

• "uri" (string) Value of the url to query for connectivity checks. Defaults to $API_ENDPOINT/connectivity-check.
• "interval" (string) Interval between connectivity checks in seconds. Defaults to 3600. To disable the connectivity checks set the interval to "0".
• "response" (string). If set controls what body content is checked for when requesting the URI. If it is an empty value, the HTTP server is expected to answer with status code 204 or send no data.

The following example configures the connectivity check by passing the balenaCloud connectivity endpoint with a 5-minute interval.

"os": {
 "network" : {
  "connectivity": {
    "uri" : "https://api.balena-cloud.com/connectivity-check",
    "interval" : "300"
  }
 }
}

udevRules

An object containing one or more custom udev rules as key:value pairs.

To turn a rule into a format that can be easily added to config.json, use the following command:

cat rulefilename | jq -sR .

For example:

root@resin:/etc/udev/rules.d# cat 64.rules | jq -sR .
"ACTION!=\"add|change\", GOTO=\"modeswitch_rules_end\"\nKERNEL==\"ttyACM*\", ATTRS{idVendor}==\"1546\", ATTRS{idProduct}==\"1146\", TAG+=\"systemd\", ENV{SYSTEMD_WANTS}=\"u-blox-switch@'%E{DEVNAME}'.service\"\nLBEL=\"modeswitch_rules_end\"\n"

The following example contains two custom udev rules that will create /etc/udev/rules.d/56.rules and /etc/udev/rules.d/64.rules. The first time rules are added, or when they are modified, udevd will reload the rules and re-trigger.

"os": {
 "udevRules": {
  "56": "ENV{ID_FS_LABEL_ENC}==\"resin-root*\", IMPORT{program}=\"resin_update_state_probe $devnode\", SYMLINK+=\"disk/by-state/$env{BALENA_UPDATE_STATE}\"",
  "64" : "ACTION!=\"add|change\", GOTO=\"modeswitch_rules_end\"\nKERNEL==\"ttyACM*\", ATTRS{idVendor}==\"1546\", ATTRS{idProduct}==\"1146\", TAG+=\"systemd\", ENV{SYSTEMD_WANTS}=\"u-blox-switch@'%E{DEVNAME}'.service\"\nLBEL=\"modeswitch_rules_end\"\n"
 }
}

sshKeys

(Array) An array of strings containing a list of public SSH keys that will be used by the SSH server for authentication.

"os": {
 "sshKeys": [
  "ssh-rsa AAAAB3Nza...M2JB balena@macbook-pro",
  "ssh-rsa AAAAB3Nza...nFTQ balena@zenbook"
 ]
}

installer

An object that configures the behaviour of the balenaOS installer image.

secureboot

(boolean) Opt-in to installing a secure boot and encrypted disk system for supported device types.

"installer": {
  "secureboot": true
}

migrate

An object that configures the behaviour of the balenaOS installer migration module.

migrate.force

(boolean) Forces the migration to run. By default the migration only runs if the installer is booting in a single disk system or the migrate argument is passed in the kernel command line.

"installer": {
  "migrate": {
    "force": true
  }
}

target_devices

(string) Overrides the default list of provisioning target mediums. May contain one or more devices, separated by spaces. The first one found will be used.

"installer": {
  "target_devices":"nvme0n1 sda"
}

Yocto version support

The following Yocto versions are supported:

• Kirkstone (4.0): Long Term Support
• Honister (3.4): EOL
• Dunfell (3.1): Long Term Support
• Warrior (2.7): EOL
• Thud (2.6): EOL
• Sumo (2.5): EOL
• Rocko (2.4): EOL
• Pyro (2.3): EOL