How to fix a python "pyverbs" application with RDMA write getting a not acknowledge (NACK)

Question

Im currently working on a RoCE (RDMA over Converged Ethernet) python application with the library pyverbs. First, i want to do a simple loopback test with an RDMA Write. I tested the setup with ib_write_bw from perftest, which worked like a charm.

This is my setup:

• OS: Ubuntu 20.04.05 LTS
• Kernel: 5.15.0-56-generic
• NIC: Mellanox ConnectX-5 MC516A-GCA_Ax 50Gbe dual-port QSFP28

I'm developing the application with a jupyter notebook. The ports are connected together with a QSFP28 cable. I set up a "client" and "server" on the same system. Both use one port of the NIC. The client performs the "RDMA Write" action. In the future, metadata will be exchanged over tcp, but for ease of debugging i exchange metadata locally in the same notebook.

Now i was able to perform an "RDMA Write" action and capture the packets. Captured RDMA packets

I keep getting not acknowledeges (NACK) from the "server". The RDMA Write packet look correct to me. It got the right payload and the headers are the same as i configured (i can post it if it would help).

I got three ideas in my head, why it would work.

1. wrong server memory adress/rkey used for rdma write
2. missing flags at server memory allocation
3. missing/wrong flags at server queue pair modification

I tried all different combinations of flags and values at the queue pair modification and memory allocation.

I start the NIC in the shi ell with

sudo mst start

Then i run my python application. I posted some codesnipped below, which i am not sure i implemented those right.

Server and client qp modification init to RTR

gid_index = 0
port_num = 1

server_attr.qp_state = e.IBV_QPS_RTR
server_attr.path_mtu = e.IBV_MTU_4096
server_attr.rq_psn = 0
server_attr.min_rnr_timer = 12
server_attr.max_dest_rd_atomic = 10
server_attr.dest_qp_num = client_qp.qp_num
server_attr.qp_access_flags = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_READ | e.IBV_ACCESS_REMOTE_WRITE
server_gr = GlobalRoute(dgid=client_ctx.query_gid(port_num=port_num,index=gid_index), sgid_index=gid_index)
server_ah_attr = AHAttr(gr=server_gr, is_global=1, port_num=1)
server_attr.ah_attr = server_ah_attr

server_qp.to_rtr(server_attr)

client_attr.qp_state = e.IBV_QPS_RTR
client_attr.path_mtu = e.IBV_MTU_4096
client_attr.rq_psn = 0
client_attr.min_rnr_timer = 12
client_attr.max_dest_rd_atomic = 10
client_attr.dest_qp_num = server_qp.qp_num
client_attr.qp_access_flags = e.IBV_ACCESS_LOCAL_WRITE | e.IBV_ACCESS_REMOTE_READ | e.IBV_ACCESS_REMOTE_WRITE
client_gr = GlobalRoute(dgid=server_ctx.query_gid(port_num=port_num,index=gid_index), sgid_index=gid_index)
client_ah_attr = AHAttr(gr=client_gr, is_global=1, port_num=port_num)
client_attr.ah_attr = client_ah_attr

client_qp.to_rtr(client_attr)

Client qp modification RTR to RTS

client_attr.qp_state = e.IBV_QPS_RTS
client_attr.timeout = 14
client_attr.retry_cnt = 7
client_attr.rnr_retry = 7
client_attr.sq_psn = 0
client_attr.max_rd_atomic = 10

RDMA Write instruction to client

client_sge = SGE(client_mr.buf,len(SEND_STRING),client_mr.lkey)
send_wr = pwr.SendWR(num_sge = 1, sg = [client_sge],opcode=e.IBV_WR_RDMA_WRITE)
send_wr.set_wr_rdma(rkey = server_mr.rkey, addr = server_mr.buf)
client_qp.post_send(send_wr)

sleep(1)
print(server_mr.read(len(SEND_STRING),0))

If there is someone with knowledge in RMDA/RoCE/Pyverbs, i would be glad for some help. I don't have any prior knowledge in those topics. This is why i have choosen to write an application in python. I have knowledge in C, but python is for me much more convient for prototyping :)

Thank for your help!