0

I am trying to create a QueuePair with ibv_create_qp() and I have to describe the size of the Queue Pair by setting the fields of the struct ibv_qp_cap and providing it to the create function. My issue is with the max_send_wr field which corresponds to the maximum number of outstanding Work Requests that can be posted to the Send Queue of that specific Queue Pair.

According to various web resources [1,2], you are supposed to set this specific field somewhere in the range of [0 ... dev_cap.max_qp_wr] with max_qp_wr being the maximum number of outstanding work requests on any Send or Receive Queue supported by the RDMA device (information you can obtain by querying your device with ibv_devinfo -v). Whilst trying to create the QueuePair, when I am setting the field to the maximum possible value (max_qp_wr which for my device is 32K), the ibv_create_qp() function fails with an ENOMEM error. By means of trial and error, I have found out that it works fine if I reduce it to 8K.

Thus, I am interested in knowing if there a deterministic way of calculating the max_send_wr value for an RDMA device without resorting to trial and error? Furthermore, according to some [1], for a specific RDMA device, there may be other considerations for the Send or Receive Queue that prevents a QP to be created with the maximum values. What are could those RDMA device-specific considerations be and how do they affect the calculation of the max value?

[1] https://www.rdmamojo.com/2012/12/21/ibv_create_qp/

[2] https://linux-rdma.vger.kernel.narkive.com/rkR0gUjT/rdma-create-qp-and-max-send-wr

kfertakis
  • 127
  • 1
  • 5

2 Answers2

1

I'm not sure there's a deterministic and portable way to answer your question. In some devices, the device pose a limit on the work queue size, rather than the number of entries (e.g., Mellanox devices), so depending on the other parameters such as the maximum inline size and scatter-gather entries, the actual number of send work requests may.

haggai_e
  • 4,689
  • 1
  • 24
  • 37
  • So when creating an RDMA application for let's say Mellanox devices, what would be the best way (best practice) to set the `max_send_wr` field in order to avoid the `ENOMEM` error. Should I calculate beforehand the maximum amount of entries that my application can post to the Send Queue of that QP? Thanks – kfertakis Dec 02 '19 at 18:37
  • 1
    I think the best practice is to set the queue size based on the requirements of the application and the properties of the network. For instance, to achieve line rate with a single queue you would want the queue size to be around the network's bandwidth-delay product. Some protocols have applications level credits and then you don't need more queue entries than the number of credits. If you wish to automatically detect which queue sizes will work you could try with a large number and then scale it down if you fail. – haggai_e Dec 04 '19 at 06:20
0

Although this is a question from two years ago, I just want to point out that when you specify a number in max_send_wr, the device driver would try to allocate a size larger than that(mostly a number in power of 2). I would suggest you dig a little bit in the driver code to find out how this calculation is happening.

In practice, if you really want to reach the maximum size you can set for the QP, you can first test the maximum max_send_wr you can set, and then add the IBV_EXP_QP_CREATE_IGNORE_SQ_OVERFLOW or IBV_EXP_QP_CREATE_IGNORE_RQ_OVERFLOW, so that you can still post WR after the max_send_wr limit.

Yupeng Tang
  • 663
  • 6
  • 7