Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How can I change the parameter for Soft RoCE (rxe)? #78

Open
shiroyd opened this issue Feb 19, 2020 · 0 comments
Open

How can I change the parameter for Soft RoCE (rxe)? #78

shiroyd opened this issue Feb 19, 2020 · 0 comments

Comments

@shiroyd
Copy link

shiroyd commented Feb 19, 2020

Hi,
I'm using Soft RoCE running on regular Broadcom 1G on-board NIC, and trying to connect to NVMe over RoCEv2 target. I have two types of NVMe/RoCEv2 target and one Linux software base NVMe/RoCEv2 target works fine so my Soft-RoCE setup should be OK but other NVMe/RoCEv2 target doesn't work like below

% nvme discover -t rdma -a 192.168.1.4 -s 4420
Failed to write to /dev/nvme-fabrics: Connection timed out

NVMe/RoCEv2 target device supplier is saying, it could be because Soft-RoCE specifies 0x80 in Responder Resources in ConnectRequest packet. It can only supports up to 64 (0x40). So if I can change the parameter for Responder Resources for ConnectRequest packet in SoftRoCE, it's likely that connection to this target works fine.
I tried to set parameter from nvme-cli like below but it doesn't change anything on wireshark.

nvme discover -t rdma -a -s 4420 -Q 64
nvme discover -t rdma -a -s 4420 -i 64

And target supplier is saying it should be parameter in Soft-RoCE.
Based on the following website, there're some parameters that we can set, I think it should one of parameter like below.
http://manpages.ubuntu.com/manpages/bionic/man7/rxe.7.html
/sys/module/rdma_rxe/parameters/max_qp_wr
but somehow I was not able to set/change parameter for it.
It doesn't allow me to edit file even I change the access privilege of directory.
root@xxxxxxx:/sys/module/rdma_rxe/parameters# chmod 777 .
root@xxxxxxx:/sys/module/rdma_rxe/parameters# ls -la
total 0
drwxrwxrwx 2 root root 0 Feb 12 04:57 .
drwxr-xr-x 6 root root 0 Feb 12 04:57 ..
--w------- 1 root root 4096 Feb 6 09:43 add
--w------- 1 root root 4096 Feb 12 04:57 remove
root@ xxxxxxx:/sys/module/rdma_rxe/parameters# touch max_qp
touch: cannot touch 'max_qp': Permission denied

So I'd like to know how I can modify parameters for Soft-RoCE.
And if possible I'd like to know how I can change Responder Resources field in ConnectRequest packet using Soft-RoCE (I could be wrong to change parameter for rdma_rxe...)

I attached screenshot of wireshark packet.
You can see "Response Resources" is set to 0x80 for ConnectRequest packet and
NVMe/RoCEv2 target device supplier is saying it needs to be 0x40 (64byte) or less.

Screen Shot 2020-02-12 at 10 05 12

Thanks in advance,
Shiroyd

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant