High loss rate on RAN550

[aibtw]

Issue Description

We are trying to run RAN550 in 2x2 mode with srsRAN gNB, and experienced high loss and low values of SINR.

Setup Details

• System: Ubuntu 22.04
• Working on i9-13900k with 128GB RAM.
• srsRAN commit: tried both bcf941b and 374200d
• RU: RAN550 v0.8.1
• UE: the one in this issue is Galaxy A14 5G but same behavior noticed on some other models as well
• Configuration file: sharing it below

Expected Behavior

stable connection with acceptable loss rate, sinr, and mcs

Actual Behaviour

Fluctuations in SINR/MCS (specifically in DL, it goes very low), and high loss. See the image (iq-scaling 5.5):

This was on iperf (UDP). Same behavior happens with iperf3 and tcp.
For example, the below is ipert tcp and I increased the iq-scaling to 7.5, almost same behavior (and same if we reduce it below 5.5)

Notice how the brate was ~2M then jumped to 4M and the SINR improved, I don't know why, the phone was stationary and iperf running.

Steps to reproduce the problem

1. configure the RAN550 like the srs guide
2. configure srsRAN as following:

amf:
  addr: 127.0.1.100                                               # The address or hostname of the AMF.
  bind_addr: 127.0.0.1                                            # A local IP that the gNB binds to for traffic from the AMF.


ru_ofh:
  ru_bandwidth_MHz: 100                # RU instantaneous bandwidth.
  t1a_max_cp_dl: 500                                              # Maximum T1a on Control-Plane for Downlink in microseconds.
  t1a_min_cp_dl: 250                                              # Minimum T1a on Control-Plane for Downlink in microseconds.
  t1a_max_cp_ul: 465                                              # Maximum T1a on Control-Plane for Uplink in microseconds.
  t1a_min_cp_ul: 250                                              # Minimum T1a on Control-Plane for Uplink in microseconds.
  t1a_max_up: 250                                                 # Maximum T1a on User-Plane in microseconds.
  t1a_min_up: 80                                                  # Minimum T1a on User-Plane in microseconds.
  ta4_max: 500                                                    # Maximum Ta4 on User-Plane in microseconds.
  ta4_min: 25                                                     # Minimum Ta4 on User-Plane in microseconds.
  is_prach_cp_enabled: false            # Configures if Control-Plane messages should be used to receive PRACH messages.
  #is_dl_broadcast_enabled: false            # Optional BOOLEAN (0). Sets downlink broadcast enabled flag. Supported: [0, 1].  (This is a workaround for SISO, set to false or commenct it for mimo)
  #ignore_ecpri_payload_size: 1     # Optional BOOLEAN (0). Sets whether or not to ignore eCPRI payload size field value. Supported [0. 1].
  compr_method_ul: bfp                 # Uplink compression method.
  compr_bitwidth_ul: 9                 # Uplink IQ samples bitwidth after compression.
  compr_method_dl: bfp                 # Downlink compression method.
  compr_bitwidth_dl: 9                 # Downlink IQ samples bitwidth after compression.
  compr_method_prach: bfp              # PRACH compression method.
  compr_bitwidth_prach: 9              # PRACH IQ samples bitwidth after compression.
  enable_ul_static_compr_hdr: true     # Configures if the compression header is present for uplink User-Plane messages (false) or not present (true).
  enable_dl_static_compr_hdr: true     # Configures if the compression header is present for downlink User-Plane messages (false) or not present (true).
  iq_scaling: 5.5                      # IQ samples scaling factor applied before compression, should be a positive value smaller than 10.
  cells:
    - network_interface: enp1s0f1np1      # Ethernet interface name used to communicate with the RU.
      ru_mac_addr: 70:b3:d5:e1:5d:a9   # RU MAC address.
      du_mac_addr: 50:7c:6f:52:85:f1   # DU MAC address.
      vlan_tag: 1                      # VLAN tag value.
      prach_port_id: [4, 5]               # PRACH eAxC port value.
      dl_port_id: [0, 1]                 # Downlink eAxC port values.
      ul_port_id: [0, 1]                 # Uplink eAxC port values.
      
cell_cfg:
  dl_arfcn: 637212                                                # ARFCN of the downlink carrier (center frequency).
  band: 78                                                        # The NR band.
  channel_bandwidth_MHz: 100                                       # Bandwith in MHz. Number of PRBs will be automatically derived.
  common_scs: 30                                                  # Subcarrier spacing in kHz used for data.
  plmn: "99970"                                                   # PLMN broadcasted by the gNB.
  tac: 7                                                          # Tracking area code (needs to match the core configuration).
  pci: 1                                                          # Physical cell ID.
  nof_antennas_dl: 2                                             # Downlink number of antennas.
  nof_antennas_ul: 2                                              # Downlink number of antennas.
  prach:
    prach_config_index: 7                                          # PRACH configuration index.
    prach_root_sequence_index: 1                                  # PRACH root sequence index.
    zero_correlation_zone: 0                                      # Zero correlation zone.
    prach_frequency_start: 0
log:
  filename: /tmp/gnb.log                                          # Path of the log file.
  all_level: debug                                              # Logging level applied to all layers.
  #mac_level: debug

pcap:
  mac_enable: true                                               # Set to true to enable MAC-layer PCAPs.
  mac_filename: /tmp/gnb_mac.pcap                                 # Path where the MAC PCAP is stored.
  mac_type: dlt
  ngap_enable: true                                              # Set to true to enable NGAP PCAPs.
  ngap_filename: /tmp/gnb_ngap.pcap                               # Path where the NGAP PCAP is stored.
  gtpu_filename: /tmp/gnb_gtpu.pcap     # Optional TEXT (/tmp/gnb_gtpu.pcap). Path for GTPU PCAPs. 
  gtpu_enable: true                    # Optional BOOL (false). Enable/disable GTPU packet capture.
  rlc_enable: true
  rlc_filename: /tmp/gnb_rlc.pcap
  f1ap_filename: /tmp/gnb_f1ap.pcap
  f1ap_enable: true
  

Please find logs and PCAPs here. All generated while running iperf3 TCP traffic, and with iq-scaling of 5.5:
gnb.log
gnb_mac.pcap
gnb_ngap.pcap
gnb_gptu.pcap
gnb_f1ap.pcap
gnb_rlc.pcap

[aibtw]

Update and more details:

We upgraded to RAN550 V1.0.4, still facing the same issue:

Doing iperf on the downlink makes the loss a little better:

But it is still inconsistent, and the loss shoots up frequently.

New configuration file:
gnb_RAN550v1_srs.txt

RU set to 20MHz, frequency adjusted to 3.6GHz, and set to MIMO 4x4 mode.
At "warning" level of logs, I can see couple of messages repeating:

2024-03-27T19:08:41.850434 [OFH     ] [W] Potentially lost '1' messages sent by the RU
2024-03-27T19:09:11.207036 [OFH     ] [W] Missed incoming User-Plane PRACH messages for slot '279.19' and sector#0
2024-03-27T19:09:11.207042 [OFH     ] [W] Missed incoming User-Plane uplink messages for slot '279.19', symbol '9' and sector#0
2024-03-27T19:09:16.650447 [OFH     ] [W] Potentially lost '1' messages sent by the RU
2024-03-27T19:09:16.705013 [SCHED   ] [W] [  830.15] UL HARQ rnti=0x460a cell=0 h_id=4: Discarding HARQ. Cause: HARQ-ACK wait timeout (256 slots) was reached, but there are still missing HARQ-ACKs and none of the received ones are positive.

gnb.log

Pcaps:
https://drive.google.com/drive/folders/1MSPd8TEAv-Bdkhr7sXkZ3nfQbdU-vKaL?usp=sharing

I would also add that on the RU side, when running kpi.sh, all packets appear to arrive on time:

Regarding synchronization, we are doing it in LLS-C1 topology (i.e. direct link between RU and DU, running ptp on both sides, while DU is synchronized to GNSS signals).
The offset we are getting is very low (less than 5ns on DU on both phc2sys and ts2phc, and <5ns on the RU as well)

I would really appreciate the help.

[andrepuschmann]

Hey @aibtw - sorry for the delayed response. I've noticed you solved the issue in the meantime using the newer firmware and updated configs as you reported in #487 (comment). I am going to close the issue therefore. Please confirm. Thanks

[aibtw]

Hi @andrepuschmann , would you mind opening the issue again? As you might have misinterpreted my other reply, where I said "However, there is still high loss rate on both uplink and downlink in all the configurations we tried", indicating that we are still facing the problem even with v1.0.4

Would really appreciate if some of you guys can have a look at my reply above.
Thanks.

[andrepuschmann]

Hi @aibtw - we are still looking at the RAN550 ourselves here but have already seen reasonable results. Could you comment on your setup a bit more please? Is it wired? What modem are you using?

Have you tried with LLS-C3 sync already? LLS-C1 is not straight-forward to get right.

A few notes on the config, please try to set:

ru_ofh:
  iq_scaling: 10
cell_cfg:
  pdsch/pusch:
    olla_target_bler: 0.1

[aibtw]

Hi @aibtw - we are still looking at the RAN550 ourselves here but have already seen reasonable results. Could you comment on your setup a bit more please? Is it wired? What modem are you using?

Have you tried with LLS-C3 sync already? LLS-C1 is not straight-forward to get right.

A few notes on the config, please try to set:

ru_ofh:
  iq_scaling: 10
cell_cfg:
  pdsch/pusch:
    olla_target_bler: 0.1

The setup is not wired, we aren't using a standalone modem. We are testing with over-air connection, using multiple android phones like OnePlus Nord N30 5G and realme 9i 5G, all of which work fine with SDRs.

For the sync type, we tried C1 and C4, with similar behavior. We still haven't tried C3 though, still trying to get a suitable GM/switch.
Do you suspect this behavior to be a sync issue that could be solved solely by moving to C3?

[aibtw]

Hey @andrepuschmann
quick update, we just tried LLS-C3, unfortunately we're still facing the same behavior that we saw in c1 and c4.

[ismagom]

Hi @aibtw , could you give us an update on this issue? Several users have been able to successfully use the RAN550.

[aibtw]

Hi @ismagom,
We are still trying to understand this issue. We tried different configs combinations, but none solved the issue until now.

Have other users you guys talked to, reported normal behavior for the UL BLER? cause that's our main issue.

we'll be grateful If you could point us to which combination of RU-DU configurations that solved the issue for other users.

[ismagom]

Can you try following the guide here: https://docs.srsran.com/projects/project/en/latest/tutorials/source/oranRU/source/rus/r550.html

I think you should also upgrade the RU firmware version

[aibtw]

We are already at v1.0.4 as noted on my reply #513 (comment), is there a newer FW version?

We also followed the guide, with a small modification. I can get UEs to connect only if I set the RU to short PRACH format. Setting it to long format (and other prach configs) as in the guide will result in "Missed incoming User-Plane PRACH messages" warning repeating indefinitely in gnb logs with no other messages, and UEs fail to connect.

So I just set it to short format as it was previously suggested by #487 (comment), and followed the rest of the guide, Is that a critical change that could affect BLER? Because we still see similar behavior as in #513 (comment). UL BLER is at 30% on average, and shoots up frequently to 60-70%.