Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sam0_eth: wait until the current tx buffer is no longer in use #20534

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

benpicco
Copy link
Contributor

@benpicco benpicco commented Apr 2, 2024

Contribution description

When many (>4) packets are sent in rapid succession, the 5th packet will overwrite the first packet in the transmit buffer before it can be sent out.

This happens e.g. when sending large (> 4500 byte) fragmented packets.

As a solution, always set the USED bit:

must be zero for the GMAC to read data to the transmit buffer. The GMAC sets this to one for the
first buffer of a frame once it has been successfully transmitted. Software must clear this bit before the buffer
can be used again.

This however is an incomplete fix: We clear the bit on each frame to be sent, if we sent the fifth frame before any of the frames has been sent, we still overwrite the first frame.

Strangely this instead disrupts transmission of non-fragmented packets afterwards, they are stuck in the tx queue.

Testing procedure

UDP Command
#include "net/sock/udp.h"

static void _fill_buffer(void *buffer, size_t len)
{
    void *end = (char *)buffer + len;
    uint16_t *b16 = buffer;
    uint16_t count = 0;
    while ((void *)b16 < end) {
        *b16++ = count++;
    }
}

static int _udp_cmd(int argc, char **argv)
{
    if (argc < 2) {
        goto usage;
    }

    int len = atoi(argv[1]);
    if (!len) {
        goto usage;
    }

    if (len & 1) {
        ++len;
    }

    static uint8_t _buf[65536];
    if (len > (int)sizeof(_buf)) {
        puts("too big");
        return -ENOBUFS;
    }

    const sock_udp_ep_t remote = {
         .family = AF_INET6,
         .addr = IPV6_ADDR_ALL_NODES_LINK_LOCAL,
         .port = 1234,
    };

    _fill_buffer(_buf, len);
    int res = sock_udp_send(NULL, _buf, len, &remote);
    if (res > 0) {
        printf("%d bytes sent\n", res);
    } else {
        printf("error: %d\n", -res);
    }

    return 0;

usage:
    printf("usage: %s <bytes>\n", argv[0]);
    return 1;
}

SHELL_COMMAND(udp2, "send large UDP packets", _udp_cmd);
Compile Flags
USEMODULE += sock_udp
USEMODULE += gnrc_ipv6_ext_frag
USEMODULE += gnrc_netif_single
CFLAGS += -DCONFIG_GNRC_PKTBUF_SIZE=65536
Receiver App (Linux)
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdlib.h>

#define MAXBUF 65536

struct adc_val {
    uint16_t u;
    uint16_t i;
};

int main()
{
    struct sockaddr_in6 sin6 = {
        .sin6_port = htons(1234),
        .sin6_family = AF_INET6,
        .sin6_addr = in6addr_any,
    };

    static char buffer[MAXBUF];
    int sock = socket(PF_INET6, SOCK_DGRAM,0);

    if (bind(sock, (struct sockaddr *)&sin6, sizeof(struct sockaddr_in6)) < 0) {
        perror("bind");
        exit(1);
    }

    int len;
    do {
        len = recv(sock, buffer, sizeof(buffer), 0);
        printf("got %d bytes\n", len);
    } while (len > 0);

    close(sock);
    return 0;
}

normal operation before fragmented packet

2024-04-02 23:10:10,929 # > ping 2001:9e8:1414:3e00:1ded:24db:24f:c127
2024-04-02 23:10:10,938 # 12 bytes from 2001:9e8:1414:3e00:1ded:24db:24f:c127: icmp_seq=0 ttl=255 time=1.082 ms
2024-04-02 23:10:11,937 # 12 bytes from 2001:9e8:1414:3e00:1ded:24db:24f:c127: icmp_seq=1 ttl=255 time=0.357 ms
2024-04-02 23:10:12,937 # 12 bytes from 2001:9e8:1414:3e00:1ded:24db:24f:c127: icmp_seq=2 ttl=255 time=0.375 ms
2024-04-02 23:10:12,937 # 
2024-04-02 23:10:12,942 # --- 2001:9e8:1414:3e00:1ded:24db:24f:c127 PING statistics ---
2024-04-02 23:10:12,947 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-02 23:10:12,951 # round-trip min/avg/max = 0.357/0.604/1.082 ms

fragmented packet is transmitted

2024-04-02 23:10:15,070 # > udp 16384
2024-04-02 23:10:15,074 # 16384 bytes sent
sending is disrupted afterwards
2024-04-02 23:10:17,433 # > ping 2001:9e8:1414:3e00:1ded:24db:24f:c127
2024-04-02 23:10:18,175 # 12 bytes from 2001:9e8:1414:3e00:1ded:24db:24f:c127: icmp_seq=0 ttl=255 time=734.203 ms
2024-04-02 23:10:18,183 # 12 bytes from 2001:9e8:1414:3e00:1ded:24db:24f:c127: icmp_seq=0 ttl=255 time=742.017 ms (DUP!)
2024-04-02 23:10:18,192 # 12 bytes from 2001:9e8:1414:3e00:1ded:24db:24f:c127: icmp_seq=0 ttl=255 time=750.437 ms (DUP!)
2024-04-02 23:10:20,433 # 
2024-04-02 23:10:20,438 # --- 2001:9e8:1414:3e00:1ded:24db:24f:c127 PING statistics ---
2024-04-02 23:10:20,444 # 3 packets transmitted, 1 packets received, 2 duplicates, 66% packet loss
2024-04-02 23:10:20,449 # round-trip min/avg/max = 734.203/742.219/750.437 ms

This is still an improvement over master where the fragmented packet is never received on the destination.

Issues/PRs references

@benpicco benpicco added Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors) State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet labels Apr 2, 2024
@github-actions github-actions bot added Platform: ARM Platform: This PR/issue effects ARM-based platforms Area: cpu Area: CPU/MCU ports labels Apr 2, 2024
@benpicco benpicco requested a review from dylad April 2, 2024 21:13
@@ -185,6 +185,7 @@ static void _init_desc_buf(void)
/* Initialize TX buffer descriptors */
for (i=0; i < ETH_TX_BUFFER_COUNT; i++) {
tx_desc[i].address = (uint32_t) tx_buf[i];
tx_desc[i].status = DESC_TX_STATUS_USED;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why setting TX buffers to one here ? This is suppose to be done by the IP. Software should only clears this bit.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bit gets set by the hardware when it's done with the buffer.
So this will mark all buffers as unused to the driver.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But GMAC IP will not find an available buffer when it want to send data if they are all marked as used already.
GMAC IP is supposed to take an unused buffer (mark == 0) then take it (mark it to one to tell software this buffer is being used)
This is not software job to set it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Screenshot_20240403_121021

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bit is set to 0 before starting the transmission.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I missed the inversion in the if statement for the thread_yield()
That's a bit weird but it makes sense.

So do we want really want to block this or drop the new frames (like this is the case for RX) ?

cpu/sam0_common/periph/eth.c Show resolved Hide resolved
@dylad
Copy link
Member

dylad commented Apr 5, 2024

@benpicco could you describe a bit more your setup ?
I tried lastest master on same54-xpro with examples/gnrc_networking.
I added your UDP command + receiver app on Linux. Desktop and same54-xpro are connected on a switch.
I was able to send UDP data then ping my computer afterwards.

@benpicco
Copy link
Contributor Author

benpicco commented Apr 5, 2024

Huh, my setup is just like that. When I send a 16384 byte UDP packet on master, I receive a broken_fragment.pcapng.gz and afterwards ping won't work anymore.

image

@dylad
Copy link
Member

dylad commented Apr 6, 2024

On my side with same54_xpro (lastest master) with gnrc_networking
I've added the required CFLAG and USEMODULE to gnrc_networking Makefile.
I've added udp2 cmd to sys/shell/cmd/gnrc_udp.c

2024-04-06 11:01:50,582 # RIOT network stack example application
2024-04-06 11:01:50,584 # All up, running the shell now
> 2024-04-06 11:06:47,820 # NETOPT_TX_END_IRQ not implemented by driver
2024-04-06 11:06:47,828 # main(): This is RIOT! (Version: 2024.04-devel-611-gdf16f45-pr/tests/sys/shell/fix_ci_error_llvm)
2024-04-06 11:06:47,832 # RIOT network stack example application
2024-04-06 11:06:47,834 # All up, running the shell now
ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:06:56,253 # ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:06:56,259 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=0 ttl=64 time=0.311 ms
2024-04-06 11:06:57,260 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=1 ttl=64 time=0.363 ms
2024-04-06 11:06:58,260 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=2 ttl=64 time=0.368 ms
2024-04-06 11:06:58,260 # 
2024-04-06 11:06:58,264 # --- fe80::d250:99ff:fe61:79b7 PING statistics ---
2024-04-06 11:06:58,269 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-06 11:06:58,273 # round-trip min/avg/max = 0.311/0.347/0.368 ms
> udp2
2024-04-06 11:07:01,504 # udp2
2024-04-06 11:07:01,505 # usage: udp2 <bytes>
> udp2 16384
2024-04-06 11:07:07,061 # udp2 16384
2024-04-06 11:07:07,066 # 16384 bytes sent
ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:07:10,576 # ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:07:10,583 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=0 ttl=64 time=0.302 ms
2024-04-06 11:07:11,583 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=1 ttl=64 time=0.299 ms
2024-04-06 11:07:12,583 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=2 ttl=64 time=0.310 ms
2024-04-06 11:07:12,583 # 
2024-04-06 11:07:12,587 # --- fe80::d250:99ff:fe61:79b7 PING statistics ---
2024-04-06 11:07:12,592 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-06 11:07:12,596 # round-trip min/avg/max = 0.299/0.303/0.310 ms
udp2 32768
2024-04-06 11:07:42,439 # udp2 32768
2024-04-06 11:07:42,448 # 32768 bytes sent
ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:07:45,945 # ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:07:45,952 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=0 ttl=64 time=0.380 ms
2024-04-06 11:07:46,952 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=1 ttl=64 time=0.298 ms
2024-04-06 11:07:47,952 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=2 ttl=64 time=0.363 ms
2024-04-06 11:07:47,952 # 
2024-04-06 11:07:47,956 # --- fe80::d250:99ff:fe61:79b7 PING statistics ---
2024-04-06 11:07:47,961 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-06 11:07:47,965 # round-trip min/avg/max = 0.298/0.347/0.380 ms
udp2 16384
2024-04-06 11:07:56,426 # udp2 16384
2024-04-06 11:07:56,431 # 16384 bytes sent
ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:07:59,874 # ping fe80::d250:99ff:fe61:79b7
2024-04-06 11:07:59,881 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=0 ttl=64 time=0.316 ms
2024-04-06 11:08:00,881 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=1 ttl=64 time=0.344 ms
2024-04-06 11:08:01,881 # 12 bytes from fe80::d250:99ff:fe61:79b7: icmp_seq=2 ttl=64 time=0.308 ms
2024-04-06 11:08:01,881 # 
2024-04-06 11:08:01,886 # --- fe80::d250:99ff:fe61:79b7 PING statistics ---
2024-04-06 11:08:01,891 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-06 11:08:01,895 # round-trip min/avg/max = 0.308/0.322/0.344 ms

With your linux app:

~/bin  ./udp                                                                                                   ✔ 
got 16384 bytes
got 32768 bytes
got 16384 bytes







Am I missing something ?

@dylad
Copy link
Member

dylad commented Apr 6, 2024

Same thing if I do not use fe80:: addr

2024-04-06 11:12:44,154 # udp2 16384
2024-04-06 11:12:44,159 # 16384 bytes sent
ping 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739
2024-04-06 11:12:46,961 # ping 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739
2024-04-06 11:12:46,969 # 12 bytes from 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739: icmp_seq=0 ttl=64 time=0.315 ms
2024-04-06 11:12:47,969 # 12 bytes from 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739: icmp_seq=1 ttl=64 time=0.321 ms
2024-04-06 11:12:48,969 # 12 bytes from 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739: icmp_seq=2 ttl=64 time=0.363 ms
2024-04-06 11:12:48,969 # 
2024-04-06 11:12:48,974 # --- 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739 PING statistics ---
2024-04-06 11:12:48,979 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-06 11:12:48,983 # round-trip min/avg/max = 0.315/0.333/0.363 ms
udp2 16384
2024-04-06 11:12:51,593 # udp2 16384
2024-04-06 11:12:51,598 # 16384 bytes sent
ping 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739
2024-04-06 11:12:54,159 # ping 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739
2024-04-06 11:12:54,167 # 12 bytes from 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739: icmp_seq=0 ttl=64 time=0.343 ms
2024-04-06 11:12:55,167 # 12 bytes from 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739: icmp_seq=1 ttl=64 time=0.393 ms
2024-04-06 11:12:56,167 # 12 bytes from 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739: icmp_seq=2 ttl=64 time=0.313 ms
2024-04-06 11:12:56,168 # 
2024-04-06 11:12:56,173 # --- 2a01:e0a:3bb:2920:53ef:5c7c:13e4:f739 PING statistics ---
2024-04-06 11:12:56,178 # 3 packets transmitted, 3 packets received, 0% packet loss
2024-04-06 11:12:56,182 # round-trip min/avg/max = 0.313/0.349/0.393 ms
> 

@benpicco
Copy link
Contributor Author

Huh that's odd - what does this print for you?

void cpu_print_info(void)
{
    const char *cpu[] = {
        "?",
        "?",
        "?",
        "?",
        "?",
        "?",
        "M4F",
        "?",
    };

    const char *series[] = {
        "?",
        "E51",  
        "E52",
        "E53",
        "E54",
        "?",    
        "D51",
    };

    printf("CPU: Cortex-%s\n", cpu[DSU->DID.bit.PROCESSOR]);
    printf("Series: SAM %s\n", series[DSU->DID.bit.SERIES]);
    printf("Die: %x\n", DSU->DID.bit.DIE);
    printf("Revision: %c\n", 'A' + DSU->DID.bit.REVISION);
}

@dylad
Copy link
Member

dylad commented Apr 11, 2024

On my same54-xpro:

2024-04-11 19:33:19,969 # CPU: Cortex-M4F
2024-04-11 19:33:19,969 # Series: SAM E54
2024-04-11 19:33:19,969 # Die: 0
2024-04-11 19:33:19,970 # Revision: A

@benpicco
Copy link
Contributor Author

I'm on Revision D 🤯

@dylad
Copy link
Member

dylad commented Apr 12, 2024

Usually, hardware works better on later revision :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: cpu Area: CPU/MCU ports Platform: ARM Platform: This PR/issue effects ARM-based platforms State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet Type: bug The issue reports a bug / The PR fixes a bug (including spelling errors)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants