You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi,
I'm using Soft RoCE running on regular Broadcom 1G on-board NIC, and trying to connect to NVMe over RoCEv2 target. I have two types of NVMe/RoCEv2 target and one Linux software base NVMe/RoCEv2 target works fine so my Soft-RoCE setup should be OK but other NVMe/RoCEv2 target doesn't work like below
% nvme discover -t rdma -a 192.168.1.4 -s 4420
Failed to write to /dev/nvme-fabrics: Connection timed out
NVMe/RoCEv2 target device supplier is saying, it could be because Soft-RoCE specifies 0x80 in Responder Resources in ConnectRequest packet. It can only supports up to 64 (0x40). So if I can change the parameter for Responder Resources for ConnectRequest packet in SoftRoCE, it's likely that connection to this target works fine.
I tried to set parameter from nvme-cli like below but it doesn't change anything on wireshark.
nvme discover -t rdma -a -s 4420 -Q 64
nvme discover -t rdma -a -s 4420 -i 64
And target supplier is saying it should be parameter in Soft-RoCE.
Based on the following website, there're some parameters that we can set, I think it should one of parameter like below. http://manpages.ubuntu.com/manpages/bionic/man7/rxe.7.html
/sys/module/rdma_rxe/parameters/max_qp_wr
but somehow I was not able to set/change parameter for it.
It doesn't allow me to edit file even I change the access privilege of directory.
root@xxxxxxx:/sys/module/rdma_rxe/parameters# chmod 777 .
root@xxxxxxx:/sys/module/rdma_rxe/parameters# ls -la
total 0
drwxrwxrwx 2 root root 0 Feb 12 04:57 .
drwxr-xr-x 6 root root 0 Feb 12 04:57 ..
--w------- 1 root root 4096 Feb 6 09:43 add
--w------- 1 root root 4096 Feb 12 04:57 remove
root@ xxxxxxx:/sys/module/rdma_rxe/parameters# touch max_qp
touch: cannot touch 'max_qp': Permission denied
So I'd like to know how I can modify parameters for Soft-RoCE.
And if possible I'd like to know how I can change Responder Resources field in ConnectRequest packet using Soft-RoCE (I could be wrong to change parameter for rdma_rxe...)
I attached screenshot of wireshark packet.
You can see "Response Resources" is set to 0x80 for ConnectRequest packet and
NVMe/RoCEv2 target device supplier is saying it needs to be 0x40 (64byte) or less.
Thanks in advance,
Shiroyd
The text was updated successfully, but these errors were encountered:
Hi,
I'm using Soft RoCE running on regular Broadcom 1G on-board NIC, and trying to connect to NVMe over RoCEv2 target. I have two types of NVMe/RoCEv2 target and one Linux software base NVMe/RoCEv2 target works fine so my Soft-RoCE setup should be OK but other NVMe/RoCEv2 target doesn't work like below
% nvme discover -t rdma -a 192.168.1.4 -s 4420
Failed to write to /dev/nvme-fabrics: Connection timed out
NVMe/RoCEv2 target device supplier is saying, it could be because Soft-RoCE specifies 0x80 in Responder Resources in ConnectRequest packet. It can only supports up to 64 (0x40). So if I can change the parameter for Responder Resources for ConnectRequest packet in SoftRoCE, it's likely that connection to this target works fine.
I tried to set parameter from nvme-cli like below but it doesn't change anything on wireshark.
nvme discover -t rdma -a -s 4420 -Q 64
nvme discover -t rdma -a -s 4420 -i 64
And target supplier is saying it should be parameter in Soft-RoCE.
Based on the following website, there're some parameters that we can set, I think it should one of parameter like below.
http://manpages.ubuntu.com/manpages/bionic/man7/rxe.7.html
/sys/module/rdma_rxe/parameters/max_qp_wr
but somehow I was not able to set/change parameter for it.
It doesn't allow me to edit file even I change the access privilege of directory.
root@xxxxxxx:/sys/module/rdma_rxe/parameters# chmod 777 .
root@xxxxxxx:/sys/module/rdma_rxe/parameters# ls -la
total 0
drwxrwxrwx 2 root root 0 Feb 12 04:57 .
drwxr-xr-x 6 root root 0 Feb 12 04:57 ..
--w------- 1 root root 4096 Feb 6 09:43 add
--w------- 1 root root 4096 Feb 12 04:57 remove
root@ xxxxxxx:/sys/module/rdma_rxe/parameters# touch max_qp
touch: cannot touch 'max_qp': Permission denied
So I'd like to know how I can modify parameters for Soft-RoCE.
And if possible I'd like to know how I can change Responder Resources field in ConnectRequest packet using Soft-RoCE (I could be wrong to change parameter for rdma_rxe...)
I attached screenshot of wireshark packet.
You can see "Response Resources" is set to 0x80 for ConnectRequest packet and
NVMe/RoCEv2 target device supplier is saying it needs to be 0x40 (64byte) or less.
Thanks in advance,
Shiroyd
The text was updated successfully, but these errors were encountered: