MCU crash after few thousand TCP requests

[raphaelvalentin]

Hi,

I am encountering an issue where the MCU crashes after sending 80KB of data via TCP several thousand times. Below is a summary of the problem:

After several thousand successful data transfers, the TCP connection begins to fail with an "Out-of-order" packet (that can occur randomly let say between 1000 to 15000). This eventually leads to a complete crash of the MCU, rendering the device unresponsive and requiring a manual reset (if no watchdog).

Could you please help me investigate this behavior? It seems to be related to TCP buffer management or the handling of out-of-order packets within the Ethernet library or underlying stack. After trying to print some error logs, I feel out of options at this moment.

Board: Nucleo stm32f767zi
All libraries: up-to-date

Thank you for your assistance!

• The arduino C/C++ reference code:

#include <LwIP.h>
#include <STM32Ethernet.h>

IPAddress ip(192, 168, 1, 196);
EthernetServer server(80);

void sendChunkedData(EthernetClient& client,
                     const unsigned char* data,
                     unsigned int length) {
    const unsigned int chunkSize = 256;
    for (unsigned int i = 0; i < length; i += chunkSize) {
        unsigned int remaining = length - i;
        unsigned int sizeToSend =
            (remaining < chunkSize) ? remaining : chunkSize;
        size_t sent = client.write(data + i, sizeToSend);
        if (sent != sizeToSend) {
            Serial.println("error size");
            Serial.flush();
            return;
        }
    }
}

// Generates Lorem Ipsum text into the provided buffer and returns the length of the generated string
size_t generateLoremIpsum(char *buffer, size_t sizeInBytes) {
    const char *lorem = "Lorem ipsum dolor sit amet, consectetur adipiscing elit. "
                        "Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. "
                        "Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. "
                        "Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. "
                        "Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. ";
    
    size_t loremLen = strlen(lorem);
    size_t pos = 0;

    while (pos + loremLen < sizeInBytes - 1) {
        strcpy(buffer + pos, lorem);
        pos += loremLen;
    }
    
    // Copy the remaining part to fill up the buffer if there's any space left
    if (pos < sizeInBytes - 1) {
        strncpy(buffer + pos, lorem, sizeInBytes - 1 - pos);
        pos += (sizeInBytes - 1 - pos); // Update position after the last part is copied
    }

    buffer[pos] = '\0'; // Ensure null termination
    return pos;         // Return the length of the generated string
}

const size_t myText_size = 80000;
size_t myText_length = 0;
uint8_t myText[myText_size];

void setup() {
    // Open serial communications and wait for port to open:
    Serial.begin(9600);
    while (!Serial) {
        ;  // wait for serial port to connect. Needed for native USB port only
    }
    myText_length  = generateLoremIpsum((char *) myText, myText_size);
    Ethernet.begin(ip);
    server.begin();
    Serial.print("server is at ");
    Serial.println(Ethernet.localIP());
    Serial.print("length of text: ");
    Serial.println(myText_length);
    Serial.print("sizeof of text: ");
    Serial.println(sizeof(myText));
}

void loop() {
    EthernetClient client = server.available();
    if (client) {

        bool currentLineIsBlank = true;
        // delay(5);
        while (client.connected()) {
            //delay(1);
            if (client.available()) {
                //delay(1);
                char c = client.read();

                if (c == '\n' && currentLineIsBlank) {
                    // delay(1);
                    client.println("HTTP/1.1 200 OK");
                    client.println("Content-Type: text/plain");
                    client.print("Content-Length: ");
                    client.println(myText_length);
                    client.println("Connection: close");
                    client.println();
                    sendChunkedData(client, (const unsigned char *) myText, myText_length);
                    // client.write((const char *) myText, myText_length);
                    break;
                }
                if (c == '\n') {
                    currentLineIsBlank = true;
                } else if (c != '\r') {
                    currentLineIsBlank = false;
                }
            }
        }
        delay(5);
        client.stop();
        delay(1);
    }
}

• The bash script:

#!/bin/bash

# URL of the file to download
URL="http://192.168.1.196/"

# Number of times to download the file
ITERATIONS=10000

# Loop to download the file multiple times
for ((i=1; i<=ITERATIONS; i++))
do
  echo "Downloading file $i of $ITERATIONS..."
  wget -T 30 -O "mytext.txt" $URL
  sleep 0.01

  # Check if the download was successful
  if [[ $? -ne 0 ]]; then
    echo "Download failed at iteration $i"
    break
  fi
done

echo "Download completed."

• The log file from the bash script:

2024-08-17 12:20:29 (7.42 MB/s) - ‘mytext.txt’ saved [79999/79999]

Downloading file 4163 of 10000...
--2024-08-17 12:20:29--  http://192.168.1.196/
Connecting to 192.168.1.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 79999 (78K) [text/plain]
Saving to: ‘mytext.txt’

mytext.txt                          100%[=================================================================>]  78.12K  --.-KB/s    in 0.01s   

2024-08-17 12:20:29 (6.80 MB/s) - ‘mytext.txt’ saved [79999/79999]

Downloading file 4164 of 10000...
--2024-08-17 12:20:29--  http://192.168.1.196/
Connecting to 192.168.1.196:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 79999 (78K) [text/plain]
Saving to: ‘mytext.txt’

mytext.txt                          100%[=================================================================>]  78.12K  --.-KB/s    in 0.01s   

2024-08-17 12:20:29 (6.80 MB/s) - ‘mytext.txt’ saved [79999/79999]

Downloading file 4165 of 10000...
--2024-08-17 12:20:29--  http://192.168.1.196/
Connecting to 192.168.1.196:80... failed: Connection timed out.
Retrying.

--2024-08-17 12:21:00--  (try: 2)  http://192.168.1.196/
Connecting to 192.168.1.196:80... ^C

real	4m16.630s
user	0m31.303s
sys	1m33.087s

• The Wireshark screenshot:
  

A Wireshark capture shows that a TCP "Out-of-order" event occurs right before the MCU crashes. This event might indicate a problem in the TCP stack or network buffer handling (lwip parameters not optimized, memory corruption or race condition?).

[raphaelvalentin]

Hi,

The update to the Ethernet source files, as described below, seems to have resolved my issue (100,000 successful HTTP requests). This change was inspired by the RTT-Ethernet source code. However, I’m still not entirely sure I understand all the underlying reasons. Anyway.

-board library: STM32 based board, version 2.8.1

compared to version STM32duino Ethernet 1.4.0:

diff -r /home/raphael/Arduino/libraries/STM32duino_STM32Ethernet/src/EthernetClient.cpp /home/raphael/Arduino/libraries/STM32duino_STM32Ethernet_STABLE/src/EthernetClient.cpp
82a83
>     delay(1);
195c196,201
<     tcp_connection_close(_tcp_client->pcb, _tcp_client);
---
>     //tcp_connection_close(_tcp_client->pcb, _tcp_client);
>     if (_tcp_client->pcb == NULL) {
>       _tcp_client->state = TCP_CLOSING;
>     } else {
>       tcp_connection_close(_tcp_client->pcb, _tcp_client);
>     }    
197a204
>   _tcp_client = NULL;
diff -r /home/raphael/Arduino/libraries/STM32duino_STM32Ethernet/src/lwipopts_default.h /home/raphael/Arduino/libraries/STM32duino_STM32Ethernet_STABLE/src/lwipopts_default.h
100c100
< #define TCP_WND                 (3*TCP_MSS)
---
> #define TCP_WND                 (4*TCP_MSS)
diff -r /home/raphael/Arduino/libraries/STM32duino_STM32Ethernet/src/utility/stm32_eth.cpp /home/raphael/Arduino/libraries/STM32duino_STM32Ethernet_STABLE/src/utility/stm32_eth.cpp
1054a1055,1062
>   if (tcp->state == TCP_CLOSING) {
>     return;
>   }
>   
>   //if (tpcb->unsent || tpcb->unacked) {
>   //    return;
>   //}
> 
1055a1064
>   tcp_accept(tpcb, NULL);
1060d1068
<   tcp_accept(tpcb, NULL);
1063c1071,1073
<   tcp_close(tpcb);
---
>   if (ERR_OK != tcp_close(tpcb)) {
>     tcp_abort(tpcb);
>   }

my loop function is:

void loop() {
    EthernetClient client = server.available();
    if (client) {

        bool currentLineIsBlank = true;
        // delay(5);
        while (client.connected()) {
            //delay(1);
            if (client.available()) {
                //delay(1);
                char c = client.read();

                if (c == '\n' && currentLineIsBlank) {
                    //delay(10);
                    client.println("HTTP/1.1 200 OK");
                    client.println("Content-Type: text/plain");
                    client.print("Content-Length: ");
                    client.println(myText_length);
                    client.println("Connection: close");
                    client.println();
                    sendChunkedData(client, (const unsigned char *) myText, myText_length);
                    // client.write((const char *) myText, myText_length);
                    break;
                }
                if (c == '\n') {
                    currentLineIsBlank = true;
                } else if (c != '\r') {
                    currentLineIsBlank = false;
                }
            }
        }
        client.stop();
        delay(1);
    }
}

[fpistm]

Hi @raphaelvalentin
Could you be more precise on the source you used ? what is the Stable ?

[raphaelvalentin]

Hi,

You can find RTT-Ethernet here: RTT-Ethernet, which is a fork of the STM32duino_STM32Ethernet library. The STM32duino_STM32Ethernet_STABLE (I called it *_stable as i could call it *_debug) code is derived from the modifications I made to the STM32duino_STM32Ethernet library, as shown by the previous diff -r.

Additionally, It will be great to explore the netconn functions within LWIP for handling more complex and unpredictable TCP/IP communication scenarios (disorder packets, loss of connection, delay on disconnection, etc). There's an example (using FreeRTOS) available in the STM Cube repository on GitHub. However, I’m not deeply familiar with the low-level implementation to build an implementation.

Actually, as a feature, it will be great to implement a Debug LOG on the library, particularly on the implementation of stm32duino/LwIP and stm32duino/Ethernet . I saw an example at lwip-Arduino. This would have helped me a lot.

Thank you for your reply.