0

I want to calculate the data transferring rate of RDMA. Using ConnectX-5 whose bandwidth is declared 200Gbps. I write data transferring code with RDMA read operation. I set start timestamp just before ibv_post_send(), and set end timestamp just after I get cq whose opcode is IBV_WC_RDMA_READ. Calculating the rate, I got transferring rate at about 268.4 Tbps which surpasses the limit. I came out that getting a CQ element dose not mean the data actually mapped to local. So use memcpy copy data once and calculate time t1=end-begin; Then I use memcpy copy same size of data and calculate mere copy time t2. Rate=data_size/(t1-t2). Still, I got 185.6Gps which also surpass the limit. Question: 1. Does the way I set timestamp right? If not, how to set timestamp right or how to design experiment to calculate transferring rate. 2. What does 100Gps transferring rate, which is declared by Mellanox, mean? Is packet transferring rate or read/write/send/receive operation?

HannanKan
  • 163
  • 2
  • 14
  • I use ib_read_bw cannot reach 22.12Gps – HannanKan Mar 29 '19 at 10:16
  • 1. Notice Gbs is [Gigabit per second](https://en.wikipedia.org/wiki/Data-rate_units#Gigabit_per_second) - not confuse with Gigabyte - that is used to measure RAM and file size in OS. 2. [Here's a simple](https://github.com/petro-rudenko/the-geek-in-the-corner/tree/master/04_rdma_file_transfer_benchmark) RDMA read/write benchmark i wrote. Maybe would be helpful. Let me know if you'll have questions. – prudenko Apr 08 '19 at 12:29
  • [To calculate bandwidth](https://github.com/petro-rudenko/the-geek-in-the-corner/blob/master/04_rdma_file_transfer_benchmark/common.c#L128) you need to convert number of bytes transmitted to Gigabits and divide to number of seconds. – prudenko Apr 08 '19 at 12:38

1 Answers1

0

As suggested in the comments, a common way to test performance is via perftest, which is a set of bandwidth and latency benchmarks, specifically ib_read_bw for the READ operation in your case.

Configuration options for ib_read_bw are detailed here as well as in its -h option.

This link also refers you to the page about the perftest package.

Noich
  • 14,631
  • 15
  • 62
  • 90