packet dropped on packet forwarder sx1303 corecell

[queifaro]

Hello,
my environment:
cpu ARM
Linux OpenWRT (Kernel 2.6)
Corecell miniPCIE (sx1303)
GPS Ublox module (indoor antenna) with pps connected to the sx1303 (GPS locked and time ok)
packet forwarder on localhost (port up and down =1700) on one shell
on another shell --> net_downlink -f 865.1 -s 7 -b 125 -r 8 -t 500 -x 1 -P 1700 (1 pkt)

in packet forwarder all pkt (except sometimes the very first) result in

JSON down: {"txpk":{"imme":true,"freq":865.100000,"rfch":0,"powe":27,"modu":"LORA","datr":"SF7BW125","codr":"4/5","ipol":false,"prea":8,"ncrc":true,"nhdr":false,"size":4,"data":"AAAAAA=="}}
INFO: [down] a packet will be sent in "immediate" mode
src/jitqueue.c:131:jit_enqueue(): Current concentrator time is 400097239, pkt_type=2
src/jitqueue.c:172:jit_enqueue(): DEBUG: insert IMMEDIATE downlink, first in JiT queue (count_us=400177239)
src/jitqueue.c:109:jit_sort_queue(): sorting queue in ascending order packet timestamp - queue size:1
src/jitqueue.c:111:jit_sort_queue(): sorting queue done - swapped
src/jitqueue.c:439:jit_print_queue(): INFO: [jit] queue contains 1 packets:
src/jitqueue.c:440:jit_print_queue(): INFO: [jit] queue contains 0 beacons:
src/jitqueue.c:446:jit_print_queue(): - node[0]: count_us=400177239 - type=2
src/jitqueue.c:309:jit_enqueue(): enqueued packet with count_us=400177239 (size=4 bytes, toa=31000 us, type=2)
WARNING: --- Packet dropped (current_time=400202948, packet_time=400177239) ---

Please note that all the utility tool work fine

Please note that the same host and the same OS work fine with 1.5 reference design and https://github.com/Lora-net/lora_gateway library and https://github.com/Lora-net/packet_forwarder

Could you please provide soe suggestions in order to investigate further ?
Thanks in advance
B/R
Fabio

[queifaro]

Hello,
little update:
after increasing TX_JIT_DELAY from 40000 to 60000 packet dropped situation disappears (happen only sometimes) but always get
WARNING: a downlink was already scheduled on rf_chain 0, overwritting it...
INFO: [jit] lgw_status returned TX_SCHEDULED

[mcoracin]

Hello,
You may need to check the actual speed of the SPI writes on your platform.
The first warning you got "Packet dropped" means that it has been taken too late from the JIT queue before writing it to the sx1303 internal TX buffer.
So yes, increasing the JIT_DELAY can help, but you may need to check why you have such latency on your platform.

Then, the second warning you get, is that apparently it took too much time between the jit_dequeue and the time at which the packet has been actually written in the sx1303 TX buffer. The sx1303 will try to send packets at the timestamp it has been programmed for. If the internal counter of the sx1303 has exceeded the timestamp value when the packet arrives in the TX buffer, it will wait for the counter to roll-over, so the packet remains in the TX buffer. And it will be overwritten by the next request.

So overall it seems there are some timing issues on your platform, to be investigated.

Best regards,
Michael

[queifaro]

Hi Michael,
i forgot to mention that the corecell <-> host connection is over usb interface. So probably the problem is not speed spi related ?
Last minute update: talking to my developers they told me that our uClibc-0.9.29 libraries do not implement the clock_nanosleep function and therefore they used the syscall(__NR_clock_nanosleep, clock_id, flags, request,remain);
Could this be the timing issue?
Thanks
Best regards,
Fabio

[mcoracin]

Hi Fabio,
Ok, so, if the interface is usb, speed issues can be even worse.
On thing that you can try, is to set the DEBUG_PERF flag to 2 to add various timing measurments:

sx1302_hal/libloragw/inc/loragw_aux.h

Line 31 in 4b42025

#define DEBUG_PERF 0 /* Debug timing performances: level [0..4] */

This flag enables prints when the _meas_time_start/_meas_time_stop macros are called in the code. For example:

sx1302_hal/libloragw/src/loragw_sx1302.c

Line 2417 in 4b42025

_meas_time_start(&tm);

If it is too verbose, you'll need to tweak it.
But at least it should measure the complete time of execution of the sx1302_send() function, which is the function which configure the sx1303 for the packets to be sent.

Hope this helps.
Best regards,
Michael

[queifaro]

Hi Michael, thanks for your support.
Maintaining TX_JIT_DELAY = 60000 and setting DEBUG_PERF = 2 the sx1302_send() execution time is :

lgw_send:1339: --- IN
PERF: .. sx1302_send 10.061000 ms
PERF: lgw_send 10.331000 ms
lgw_send:1473: --- OUT

Is it what you expect (more or less) ?

a note my host processor support USB 2.0 Full-Speed 12 Mbps speed.

Thanks
Best regards,
Fabio

[mcoracin]

Hi Fabio,

Indeed, it seems that you have more latency than what I get on my side with a Raspberry Pi. I get 3ms for the lgw_send() where you have 10ms.

This means that there could be latencies here and there which make the packets reaching the sx1303 TX buffer too late for being sent on time.

The way communication between the linux host and the corecell MCU is done for USB corecells is that "SPI" commands are grouped by bulks to minimize the number of USB transactions which have, if I remember well a minimum duration of 1ms.
You can search for LGW_COM_WRITE_MODE_BULK and lgw_com_flush in the code to see how it is used to group communication with the sx1303.

In you case, it would be worth check timings you get in the downlink request chain of execution:

• jit_queue
• jit_peek
• jit_dequeue
• lgw_send => the call to lgw_com_flush in the sx1302_send() function.

The packet is supposed to be peeked from the jit_queue according to JIT_DELAY in advance to departure time.
You need to ensure that it reaches the SX1303 before the departure time.

I hope you'll be able to progress on this.

Which linux host are you using ?

Best regards,
Michael

[queifaro]

Hi Michael,
i will try to investigate following your suggestions.

My Host Linux is Microchip AT91SAM9260:

https://ww1.microchip.com/downloads/aemDocuments/documents/OTH/ProductDocuments/DataSheets/ARM_926EJS_TRM.pdf

 180 MHz ARM926EJ-S™ ARM® Thumb® Processor
̶ 8 Kbytes Data Cache, 8 Kbytes Instruction Cache, MMU
 Memories
̶ 32-bit External Bus Interface supporting 4-bank SDRAM/LPSDR, Static Memories, CompactFlash, SLC NAND
Flash with ECC
̶ Two 4-Kbyte internal SRAM, single-cycle access at system speed
̶ One 32-Kbyte internal ROM, embedding bootstrap routine
 Peripherals
̶ ITU-R BT. 601/656 Image Sensor Interface (ISI)
̶ USB Device and USB Host with dedicated On-Chip Transceiver
̶ 10/100 Mbps Ethernet MAC Controller (EMAC)
̶ One High Speed Memory Card Host
̶ Two Master/Slave Serial Peripheral Interfaces (SPI)
̶ Two 3-channel 32-bit Timer/Counters (TC)
̶ One Synchronous Serial Controller (SSC)
̶ One Two-wire Interface (TWI)
̶ Four USARTs
̶ Two UARTs
̶ 4-channel 10-bit ADC

Thanks
Best regards,
Fabio