ESP8266 - FlashBdev start sector

[git-n-pissed]

At the bottom of flashbdev.py exists the following code snippet:

size = esp.flash_size()
if size < 1024 * 1024:
    bdev = None
else:
    start_sec = esp.flash_user_start() // FlashBdev.SEC_SIZE
    if start_sec < 256:
        start_sec += 1  # Reserve space for native code

    # 20K at the flash end is reserved for SDK params storage
    bdev = FlashBdev(start_sec, (size - 20480) // FlashBdev.SEC_SIZE - start_sec)

I'm wondering about the if start_sec < 256 part. Seems like this should be <=, or maybe that entire line should not exist. The file esp8266_2m.ld specifies an irom of 1M - 36K (988K to speak in common units), with that "36K" being deduced to account for header data. 988K + 36K = 1024K = 1M which is the max size of an ESP8266's executable code region.

I am making one assumption based on my experience with MP on the ESP8266 from years ago, that being, the ESP8266 does not use or otherwise require the reserve sector (i.e. the boot sector) of the FAT file system to be in its executable code region.

[jimmo]

The esp.flash_user_start() returns the _firmware_size linker symbol which is calculated as ORIGIN(irom0_0_seg) + LENGTH(irom0_0_seg) - 0x40200000;, which will evaluate to 0x40209000 + (1M - 36K) - 0x40200000.

So this ends up including the 36k -- i.e. start_sec will be exactly equal to 256 on "2M" build.

What this actually means is that on 2M builds, there is no reserved sector -- this feature is only available on 512k and 1M builds.

So I think the code as it stands is correct? If it were changed to <= then we would be putting the reserved sector past the 1M boundary (where it would be useless). If the line didn't exist, then we wouldn't be doing a reserved sector.

(I've been working on and around MicroPython for nearly 7 years and never come across this "reserved sector" thing. Turns out it is required to use https://docs.micropython.org/en/latest/library/esp.html#esp.set_native_code_location Nothing to do with the filesystem boot sector).

[git-n-pissed]

What this actually means is that on 2M builds, there is no reserved sector -- this feature is only available on 512k and 1M builds.

So my assumption was wrong? MP does/can boot from FAT and that reserve/boot sector needs to exist in the 1M section of flash the ESP8266 has mapped for code execution? Why would this not be "available" on 2M+ builds?

So I think the code as it stands is correct? If it were changed to <= then we would be putting the reserved sector past the 1M boundary (where it would be useless). If the line didn't exist, then we wouldn't be doing a reserved sector.

(I've been working on and around MicroPython for nearly 7 years and never come across this "reserved sector" thing. Turns out it is required to use https://docs.micropython.org/en/latest/library/esp.html#esp.set_native_code_location Nothing to do with the filesystem boot sector).

Ah, but you have come across it before ;)

https://forum.micropython.org/viewtopic.php?p=38024#p38024

And the requirement that a FAT file system contain a reserve sector I guess is what I'm hung up on. How can the current code be correct if in some cases the required reserve sector isn't being accounted for?

I'm using a 4M ESP8266 (so building for the "GENERIC" 2M+ board) and using the modifications I made to makeimg.py in our old thread I cannot get the image to boot unless the reserve sector is accounted for in flashbdev.py and in the mkfs.fat command used to make the file system appended to the image after it is built.

Here is the code from makeimg.py, updated from our old thread to account for 3.5 years of changes to MP. Note too that where mkfs.fat used to use the value of reserved_secs for its -R argument, now it is set to a value of 1 always because FAT requires a reserve sector of at least 1 segment and I assumed the reserve segment (if actually used by MP) would never need to be larger that 1 sector:

import sys
import struct
import hashlib
import os

# This region at the start of flash contains a small header and then segments
# containing .text, .data and .rodata, and so must be large enough to hold all
# of this.  This data is loaded to the appropriate places in RAM by the ROM
# bootloader at boot.  After this in flash comes .irom0.text, which must begin
# on a flash erase-page boundary.
SEGS_MAX_SIZE = 0x9000

assert len(sys.argv) == 4

md5 = hashlib.md5()

with open(sys.argv[3], "wb") as fout:

    with open(sys.argv[1], "rb") as f:
        data_flash = f.read()
        fout.write(data_flash)
        # First 4 bytes include flash size, etc. which may be changed
        # by esptool.py, etc.
        md5.update(data_flash[4:])
        print("flash    ", len(data_flash))

        # Print info about segments in this first part of flash
        num_segs = struct.unpack_from("<BBBBI", data_flash, 0)[1]
        offset = 8
        for seg_num in range(num_segs):
            seg_name = [".text", ".data", ".rodata"][seg_num]
            seg_offset, seg_size = struct.unpack_from("<II", data_flash, offset)
            print(" {:7}  {} at 0x{:x}".format(seg_name, seg_size, seg_offset))
            offset += 8 + seg_size

    with open(sys.argv[2], "rb") as f:
        data_rom = f.read()

    pad = b"\xff" * (SEGS_MAX_SIZE - len(data_flash))
    assert len(pad) >= 4
    fout.write(pad[:-4])
    md5.update(pad[:-4])
    len_data = struct.pack("I", SEGS_MAX_SIZE + len(data_rom))
    fout.write(len_data)
    md5.update(len_data)
    print("padding  ", len(pad))

    fout.write(data_rom)
    md5.update(data_rom)
    print("irom0text", len(data_rom))

    fout.write(md5.digest())

    print("total    ", SEGS_MAX_SIZE + len(data_rom))
    print("md5      ", md5.hexdigest())

build_dir = os.path.dirname(sys.argv[3])
fs_files = 'fs_files'  # Location of files to copy to file system
fat_fs = os.path.join(build_dir, 'fat.fs')  # Location of file system
if os.path.isdir(fs_files):
    import apt
    import re
    import subprocess

    apt_cache = apt.Cache()
    if 'mtools' not in apt_cache or not apt_cache['mtools'].is_installed:
        print('The command "mcopy" (a part of "mtools") is required to build the file system image.  To install type "sudo apt-get install mtools".')
        sys.exit(1)

    print('Building file system image "{fat_fs}"'.format(fat_fs=fat_fs))

    # Read ELF file to determine firmware size
    cmd = 'readelf -a {build_dir}/firmware.elf'.format(build_dir=build_dir)
    proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
    out, error = proc.communicate()

    pattern = re.compile(' *\\d+: (\\d+).*_firmware_size')  # Example (without quotes): "  3453: 00100000     0 NOTYPE  GLOBAL DEFAULT  ABS _firmware_size"
    match = re.findall(pattern, out.decode())
    if match:
        flash_user_start = int(match[0], 16)  # The same value that esp.flash_user_start() returns. When irom0_0_seg = 1011712 (0xF7000) which is the max value for an ESP8266 with a 4MB flash, this will be 1048576 (0x100000).

    else:
        raise Exception('Unable to determine firmware size!')

    print('flash_user_start', flash_user_start)

    # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    #
    # Set flash_size to the size of your target flash. This will be the same value returned by esp.flash_size().
    #
    # For an ESP8266 with a 4MB flash, this will be 4194304 (0x400000)
    # For an ESP8266 with a 2MB flash, this will be 2097152 (0x200000)
    # For an ESP8266 with a 1MB flash, this will be 1048576 (0x100000)
    # For an ESP8266 with a 512KB flash, this will be 524288 (0x80000)
    #
    # !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
    if build_dir.endswith('GENERIC'):
        # TODO: How to determine difference between 2MB and 4MB boards?  Assume 4MB for now
        flash_size = 0x400000
        # flash_size = 0x200000
    elif build_dir.endswith('1M'):
        flash_size = 0x100000
    elif build_dir.endswith('512K'):
        flash_size = 0x80000
    else:
        print('Failed to determine flash size')
        sys.exit(1)

    sec_size = 4096  # Needs to match SEC_SIZE from flashbdev.py
    reserved_secs = 1 if flash_user_start // sec_size < 256 else 0  # Needs to match the value of "space reserved for native code" in flashbdev.py
    start_sec = flash_user_start // sec_size + reserved_secs  # Same formula used to determine the "start_sec" value passed to FlashBdev in flashbdev.py
    blocks = (flash_size - 20480) // sec_size - start_sec  # Same formula used to determine the "blocks" value passed to FlashBdev in flashbdev.py
    fs_start = start_sec * sec_size  # Starting location of the file system

    # Delete the old file system if it exists
    if os.path.isfile(fat_fs):
        try:
            os.remove(fat_fs)

        except Exception:
            print('Failed to delete existing file system image "{fat_fs}"'.format(fat_fs=fat_fs))
            sys.exit(1)

    # Use DD to create zeroed out file of the correct size for the file system
    cmd = 'dd if=/dev/zero of={fat_fs} count={blocks} bs={sec_size}'.format(fat_fs=fat_fs, blocks=blocks, sec_size=sec_size)
    if subprocess.call(cmd, shell=True) != 0:
        print('Failed to create file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
        sys.exit(1)

    # Use mkfs to convert the previously created file into a FAT file system
    cmd = 'mkfs.fat -F 12 -R {reserved_secs} -S {sec_size} {fat_fs}'.format(reserved_secs=1, sec_size=sec_size, fat_fs=fat_fs)
    if subprocess.call(cmd, shell=True) != 0:
        print('Failed to convert file system image "{fat_fs}" to FAT file system with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
        sys.exit(1)

    # Add files to file system
    print('Adding files to file system image "{fat_fs}"'.format(fat_fs=fat_fs))

    for root, dirs, files in os.walk(fs_files):
        target_dir = os.path.relpath(root, fs_files)

        if root != fs_files:
            cmd = 'mmd -i {fat_fs} ::{target_dir}'.format(fat_fs=fat_fs, target_dir=target_dir)
            if subprocess.call(cmd, shell=True) != 0:
                print('Failed to make directory in file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
                sys.exit(1)

        for file in files:
            source_file = os.path.join(root, file)

            if root == fs_files:
                target_file = file

            else:
                target_file = os.path.join(target_dir, file)

            cmd = 'mcopy -i {fat_fs} {source_file} ::{target_file}'.format(fat_fs=fat_fs, source_file=source_file, target_file=target_file)
            if subprocess.call(cmd, shell=True) != 0:
                print('Failed to copy file to file system image "{fat_fs}" with command "{cmd}"'.format(fat_fs=fat_fs, cmd=cmd))
                sys.exit(1)

    # Calculate padding needed between fw and fs
    fout_name = sys.argv[3]
    fout_size = os.path.getsize(fout_name)
    pad = b'\xff' * (fs_start - fout_size)

    # Append the file system to the bin file
    print('Appending file system image "{fat_fs}" to "{fout_name}"'.format(fat_fs=fat_fs, fout_name=fout_name))

    with open(fout_name, 'ab') as fout:
        if pad:
            fout.write(pad)

        with open(fat_fs, 'rb') as fat_fs_hndl:
            fout.write(fat_fs_hndl.read())

    # Give warning message if no boot.py is found in the files copied to the file system
    if not os.path.isfile(os.path.join(fs_files, 'boot.py')):
        print('Warning: Your build has a file system without a boot.py file. This will prevent your ESP8266 from booting unless you have done something to change that.')

[jimmo]

So my assumption was wrong? MP does/can boot from FAT and that reserve/boot sector needs to exist in the 1M section of flash the ESP8266 has mapped for code execution? Why would this not be "available" on 2M+ builds?

@git-n-pissed I'm not sure I follow sorry.

None of this is to do with putting the boot sector into the executable region.

It's just about ensuring that you leave enough room after the firmware but before the filesystem.

On 2M builds, the firmware takes up the whole executable range, so we just start the filesystem at the 1MiB boundary.

(I don't know the history, but this might have been an oversight, and the 2MiB linker script probably should have reserved one sector such that the firmware size was one sector smaller, and then start_sec on 2MiB builds would be 255 not 256. I'm pretty sure we could actually still fix this today without breaking anything, because the filesystem would still end up starting at the same place).

Ah, but you have come across it before ;)

That thread isn't about the feature that I was describing and what the if start_sec < 256: is for (which is, for 512k and 1M builds, being able to optionally store code generated by the native emitter in XIP flash rather than IRAM).

[git-n-pissed]

Ah, but you have come across it before ;)

That thread isn't about the feature that I was describing and what the if start_sec < 256: is for (which is, for 512k and 1M builds, being able to optionally store code generated by the native emitter in XIP flash rather than IRAM).

Sorry for the confusion. The "it" I was referring to was the "reserve sector" and I was treating the FAT reserve sector (which is what the old thread was referring to) and the sector for native code on the ESP as the same thing, but they are and they aren't.

Let me dig into this some more to figure out what is going on and come back with better data. What I know now is that my build boots and works as expected on a 4M ESP8266 if I change the < to a <=, (which causes start_sec to be 257), but that same build does not boot without that change. I also know that back when we were chatting in the old thread, start_sec also equaled 257, but that was before 3.5 years of changes to flashbdev.py.