Different behaviour on ESP32 vs ESP8266

[trullock]

I understand this may not technically be an "issue" but its a problem I've encountered and would greatly appreciate any help or pointer you can give me.

I'll reproduce my issue I posted here: https://esp32.com/viewtopic.php?f=19&t=35869&sid=86902cd8c4a9aa8cb4547f524096db9b

I'm porting some code from ESP8266 to ESP32 and I've found a behavioural difference between the two which I need to solve.

I'm writing log data to a LittleFS file on the ESP8266, calling flush after each write. On the ESP8266 if there is a power failure before file.close() is called, the data is synced to flash and can be read back. On the ESP32 it is not, the file has 0 Bytes.

I loosely understand why, its to do with how LFS stages the content and when sync() gets called. This thread is relevant littlefs-project/littlefs#344

My question is, whats (or rather where) is different between the ESP32 and ESP8266? What code or configuration settings are different between the two?

What is the ESP32 doing, why is the behaviour different and how to I get it to behave like the ESP8266?

Example code for esp8266, esp32 version has some slight syntax differences

#include <LittleFS.h>
File currentLog;

#define DELAY 200
uint64_t lastMillis = 0;
uint8_t loopBuffer[6] = {0};
uint16_t counter = 0;

void setup() {
  Serial.begin(115200);

  if (!LittleFS.begin())
  {
    Serial.println("LittleFS mount failed");
    return;
  }


  currentLog = LittleFS.open("test", "r");
  auto size = currentLog.size();
  auto rows = size > 0 ? size / 6 : 0;

  Serial.print("Opened previous log of size: ");
  Serial.print(size);
  Serial.print(" bytes, found ");
  Serial.print(rows);
  Serial.println(" rows");
  currentLog.close();

  Serial.println("Waiting 5 secs...");
  delay(5000);

  currentLog = LittleFS.open("test", "w");
  
  lastMillis = millis();
}

void loop() {
  uint64_t now = millis();
  if(now < lastMillis + DELAY)
    return;

  lastMillis = now;

  currentLog.write(loopBuffer, 6);
  currentLog.flush();
  
  Serial.print("Looped: ");
  Serial.println(counter++);

}

Thanks

[BrianPugh]

Looking at the underlying glue code, the arduino 8266 implementation and my implementation are quite similar when it comes to flush:

See:

8266:

   void flush() override {
        if (!_opened || !_fd) {
            return;
        }
        int rc = lfs_file_sync(_fs->getFS(), _getFD());
        if (rc < 0) {
            DEBUGV("lfs_file_sync rc=%d\n", rc);
        }
    }

esp32

static int vfs_littlefs_fsync(void* ctx, int fd)
{
    ....
    file = efs->cache[fd];
    res = lfs_file_sync(efs->fs, &file->file);
    sem_give(efs);
    ....
    return res;
}

This has also been discussed a bit here: #48. Some of the comments have actually been cross-posted to the other issues you linked.

In summary, it seems like there's a workaround, but it also seems like it's an upstream issue. From a quick skim, I'm not seeing a functional difference when it comes to the writing and flushing code between the arduino-8266 and this esp32 implementation. I agree that the current behavior is unintuitive and less than ideal.

A possible cause is that the configured 8266 cache sizes are smaller, so a flush/sync might be more likely to always flush to disk. I haven't had a chance to mess around with this.

[trullock]

Yeah, I've been chasing this issue across the internet for a while without coming to a full understanding or solution (yet), as is evident from the array of open issues here and there...

How do i call sync() from the Arduino ESP32 LittleFS API? Is that the workaround you speak of?

Here's the ESP32 code you can use to repro/test

#include <LittleFS.h>
File currentLog;

#define DELAY 200
uint64_t lastMillis = 0;
uint8_t loopBuffer[6] = {0};
uint16_t counter = 0;

void setup() {
  Serial.begin(115200);

  if (!LittleFS.begin(true))
  {
    Serial.println("LittleFS mount failed");
    return;
  }


  currentLog = LittleFS.open("test", FILE_READ);
  auto size = currentLog.size();
  auto rows = size > 0 ? size / 6 : 0;

  Serial.print("Opened previous log of size: ");
  Serial.print(size);
  Serial.print(" bytes, found ");
  Serial.print(rows);
  Serial.println(" rows");
  currentLog.close();

  Serial.println("Waiting 5 secs...");
  delay(5000);

  currentLog = LittleFS.open("test", FILE_WRITE, true);
  
  lastMillis = millis();
}

void loop() {
  uint64_t now = millis();
  if(now < lastMillis + DELAY)
    return;

  lastMillis = now;

  currentLog.write(loopBuffer, 6);
  currentLog.flush();
  
  Serial.print("Looped: ");
  Serial.println(counter++);

}

Thanks

[BrianPugh]

So I can't say I've used arduino-esp32, but it doesn't look like the sync functionality is exposed. As an alternative, if you set the configuration parameter CONFIG_LITTLEFS_FLUSH_FILE_EVERY_WRITE true, a flush will be internally called after every write.

[trullock]

Doesn't Arduino just sit on top of esp-idf and the associated components, which means all this library's/component's functions are available? Can I call vfs_littlefs_fsync (or whatever else I need to call) with Arduino' FS:File object? or is that where the "functionality isnt exposed" part you refer to really comes into play?

I'm happy to switch to just calling the littlefs lib directly, I'm only using the Arduino entry point from a convenience perspective from previous familiarity.

I will try the config option but it means I need to rebuild the project I'm using to be in esp-idf which is slightly more involved, ill try that in the next few days

[HamzaHajeir]

Up to my experience, even ESP8266 loses file content when a power failure happens while writing.

esp8266/Arduino#8155

[trullock]

@HamzaHajeir thanks I've seen that thread. Not only can I not reproduce that 8266 behaviour (all bytes are always written) but on the 32 no bytes are written/synced, even when exceeding the ram buffer size

[HamzaHajeir]

Haven't tested for ESP32, but I can confirm the issue, the provided log was the interesting part, as it's a very short window and it can be missed easily by the reset button push. Might it be a multi threading related issue for ESP32 or not?

…

On Fri, Sep 22, 2023, 13:35 Andrew Bullock ***@***.***> wrote: @HamzaHajeir <https://github.com/HamzaHajeir> thanks I've seen that thread. Not only can I not reproduce that 8266 behaviour (all bytes are always written) but on the 32 no bytes are written/synced, even when exceeding the ram buffer size — Reply to this email directly, view it on GitHub <#144 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3O7J7QJTEN4UUOIG53NXLX3VSYNANCNFSM6AAAAAA47WL6U4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[M-Bab]

You seem to write only little amounts of data (smaller than CONFIG_LITTLEFS_WRITE_SIZE). Did you have a look at my workaround here: littlefs-project/littlefs#344 (comment) ?

If you can try it out and write it this helps. I am not sure if this is still necessary. Of course it is inefficient to do this all the time - so it works best if you have some kind of brown out detection triggering immediately before power loss.

[trullock]

Can that workaround be invoked from LittleFS via Arduino ESP? I don't think it can, but I will try later.

I know I'm writing small data per loop, but even if you let it run for ages (to have written lots), a reset still leaves a 0B file, surely it should have written some of it...

[BrianPugh]

@trullock Have you tried @justintconroy #48 (comment) solution? I just ran a test, and for me calling

fflush(f);
fsync(fileno(f));

properly writes the files to disk when reseting prior to closing the file.

The fflush will flush the FILE buffers to littlefs buffers.
The fsync will flush littlefs buffers to disk.

[trullock]

@BrianPugh how do I invoke these with an Arduino FS::File? Is the File f in your above example? The functions seem to take an int, is this really a pointer or some internal handle?

[HamzaHajeir]

You seem to write only little amounts of data (smaller than CONFIG_LITTLEFS_WRITE_SIZE). Did you have a look at my workaround here: littlefs-project/littlefs#344 (comment) ?

If you can try it out and write it this helps. I am not sure if this is still necessary. Of course it is inefficient to do this all the time - so it works best if you have some kind of brown out detection triggering immediately before power loss.

It seems that CONFIG_LITTLEFS_WRITE_SIZE is an ESP-IDF configuration. Arduino core doesn't build on the IDF.

But anyway, this was just an MCVE. The lost files were actually larger, holding WiFi and MQTT credentials, MQTT server url, and some other global values.

[trullock]

I'm going to build Arduino from sources so I can play with that config setting when I get a moment. However, no amount of writing gets persisted to "disc". Surely this is a bug. Ram buffers and wear considerations are one thing, but LFS can't be touted as highly resilient when it never writes and bytes for me...

I'm going to write some test code that proves the WRITE_SIZE has no effect, although there's threads elsewhere that already identify this