KRCore kernel module throws an error

[snowzjx]

Dear authors,

I am using the 7ba3bf6 as indicated in your paper for evaluation.

I have successfully installed the patched OFED driver.

I have compiled the KRdmaKitSyscall.ko successfully.

However, when I tried to install the module, there are some errors:

insmod: ERROR: could not insert module KRdmaKitSyscall.ko: Operation not permitted

The dmesg has the following output:

[ 5760.285891] KRdma kernel module init start
[ 5760.287625] Fail to start KRdma kernel module

Moreover, I am confused by the gid in the Makefile in the KRdmaKit-syscall folder and have no idea which gid to fill.

Could you help me with the above problems?

Thanks!

[wxdwfc]

It seems that the RPC client's failed init (https://github.com/SJTU-IPADS/krcore-artifacts/blob/develop/KRdmaKit-syscall/src/client.rs#L143), can you add some log to see which step goes wrong?

The meta gid can be set as the same gid as the RNIC you use for inserting the krcore module.

[snowzjx]

krcore-artifacts/KRdmaKit-syscall/src/client.rs

Line 156 in 7ba3bf6

return None;

It seems the function returns here.

[wxdwfc]

It seems that no available RNIC is found on your device. Can you check that the NIC is available, i.e., what is the hardware configuration you use? You can use ibstatus to check that.

[snowzjx]

Hey, I have done some checks and found the error is raised from the query_gid: function at

krcore-artifacts/rust-kernel-rdma/KRdmaKit/src/device.rs

Line 81 in 73e3d7e

let err = unsafe { hca.query_gid(hca_ptr, port as u8, 0, &mut gid as *mut ib_gid) };

Since I am using RoCEv2, the gids seems not available from query_gid function so I tried to manually set the gid. The client can be successfully created.

However, I now got a new error:
[35236.576172] [dct] err to re-bring to rtr.
[35236.576173] [dct] err to bring to ready.

I checked the code and found the error is thrown from ib_modify_qp function and it returns an error code of -22, seems the parameters used are not correct.

Could you help me with the new problem? Thanks!

[wxdwfc]

Hi, the problem means that DCT QP has not been successfully created. Have you verified your NIC supports DCT?

Besides, I strongly recommended using IB NIC for running, I have not tested on RoCE NICs.

[snowzjx]

The strange thing is that the function bring_dc_to_init does not throw errors.

I am using Mellanox CX5 RNIC.