Issue with Accessing Full 16 MB Flash on ESP32 with MicroPython

[arunyava]

Hi everyone,

I'm using an ESP32-WROOM-32 module which has 16 MB of flash memory. However, when I check the filesystem size using the os module in MicroPython, it only shows 2 MB.

Here is the code I'm using to check the filesystem size:

import os
fs_stat = os.statvfs('/')
total_fs = fs_stat[0] * fs_stat[2]
free_fs = fs_stat[0] * fs_stat[3]
used_fs = total_fs - free_fs

print(f"Total filesystem size: {total_fs} bytes")
print(f"Used filesystem size: {used_fs} bytes")
print(f"Free filesystem size: {free_fs} bytes")
I have tried flashing the firmware with a custom partition table but still see the same issue. Below is the partition table I used:
# Name,   Type, SubType, Offset,   Size, Flags
nvs,      data, nvs,     0x9000,   0x5000,
phy_init, data, phy,     0xe000,   0x1000,
factory,  app,  factory, 0x10000,  0x100000,
vfs,      data, fat,     0x110000, 0xEF0000,

Any help on how to utilize the full 16 MB flash memory would be greatly appreciated! I am new to micropython and esp

Thanks in advance.

[glenn20]

The default partition table for the flash storage on the ESP32 WROOM images is for a 4MB flash system.

You have two options:

1. Download and run the mp-image-tool-esp32 python tool to modify the partition table on your esp32 device, eg:

   mp-image-tool-esp32 u0 -f 16M --resize vfs=0
   

   You run this on your computer to modify the partition table of the device attached to usb port (/dev/ttyUSB0). This does 2 things: it sets the size of the flash specified in the bootloader to 16 MB and then modifes the partition table so that the vfs partition takes up all the available space. This will also erase your current vfs partition, so backup any important files you have on there first.

2. OR compile your own micropython image, changing the partition table to a 16MB table by changing the relevant line in ports/esp32/boards/sdkconfig.base to:

   CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-16MiB.csv"
   

[mattytrentini]

Hmm, we should add a variant for 16MB (and 8MB I guess). I think it's just a matter of changing mpconfigboard.cmake to add a variant that picks up a different (new?) sdkconfig that configures the flash size and partition table...

[glenn20]

1. Download and run the mp-image-tool-esp32 python tool to modify the partition table on your esp32 device, eg:

   mp-image-tool-esp32 u0 -f 16M --resize vfs=0
   

   You run this on your computer to modify the partition table of the device attached to usb port (/dev/ttyUSB0). This does 2 things: it sets the size of the flash specified in the bootloader to 16 MB and then modifes the partition table so that the vfs partition takes up all the available space. This will also erase your current vfs partition, so backup any important files you have on there first.

If you really dont want to erase the existing 2MB vfs flash partition or you like to organise your flash storage differently (eg. to mitigate possible filesystem corruption if you are doing frequent writes to data files), you could alternatively create a new vfs partition to fill the left over space and create a new littlefs filesystem on that, and then mount that at runtime, eg:

mp-image-tool-esp32 --add data=fat:4M:0

Then create the new filesystem after booting into the micropython board:

import vfs
from esp32 import Partition

p = Partition.find(Partition.TYPE_DATA, label="data")
vfs.VfsLfs2.mkfs(p)
fs = vfs.VfsLfs2(p)
vfs.mount(fs, "/data")

Add the following to boot.py to mount the "data" filesystem on bootup:

import vfs
vfs.mount(Partition.find(Partition.TYPE_DATA, label="data"), "/data")

2. OR compile your own micropython image, changing the partition table to a 16MB table by changing the relevant line in ports/esp32/boards/sdkconfig.base to:

   CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-16MiB.csv"
   

If you choose option 2 you will also need to erase the existing vfs partition by erasing the flash before flashing the new firmware (eg. with esptool.py erase_flash), otherwise micropython will still see a 2MB filesystem at the start of your new 16MB partition.

[scruss]

Noting (with thanks!) that your mp-image-tool-esp32 tool seems to be easing boot problems I'd been having consistently with the SparkFun Thing Plus SkeleBoard (aka a SparkFun Thing Plus - ESP32 WROOM (U.FL) with a snazzy paint job and 16 MB of flash). A sequence of:

mp-image-tool-esp32 u1 -f 16M --resize vfs=0
mpremote u1 reset
mp-image-tool-esp32 u1 --fs grow vfs

appears to make it a reliable performer. The combination of a (now low-end) ESP32 WROOM with a large flash chip seems quite rare now, so adding special builds for it is probably not something anyone needs to do.