3

I have just configured a server-to-server 4x SDR InfiniBand link to increase bandwidth for the backend storage of my virtual machines, specifically a database server. I am using iSCSI with TGTD on CentOS 6 and I am able to achieve about 6.45Gbps using IPoIB when testing with iperf.

I have managed to tune the iSCSI initiator and target to the point where I can achieve about 260MB/s write to an SSD. However, this is still not an optimal write speed given I can write at about 500-600MB/s locally.

The parameters I am currently using are:

  • MaxRecvDataSegmentLength 4194304
  • MaxXmitDataSegmentLength 524288
  • MaxBurstLength 4194304
  • FirstBurstLength 262144
  • MaxOutstandingUnexpectedPDUs=0
  • MaxOutstandingR2T=1
  • MaxCommands=256
  • QueueDepth=128

Has anyone had any experience in tuning this? Is it possible to get any higher than this with configuration changes? Even by doing a RAID 0 on two devices, I can still only achieve the same bandwidth, which I would have expected higher.

jaseeey
  • 1,462
  • 16
  • 20

1 Answers1

2

The results you got are reasonably expectable with current setup. You can get additional perfomance gain with iser transport mode.

Configuration tips from README.iser

Target:

    ./tgtadm --lld iser --mode target \
             --op new --tid 1 --targetname "iqn.$(hostname).t1"
    ./tgtadm --lld iser --mode target \
             --op bind --tid 1 --initiator-address ALL
    ./tgtadm --lld iser --mode logicalunit \
             --op new --tid 1 --lun 1 \ --backing-store /dev/sde
             --bstype rdwr

Initiator:

    iscsiadm -m discovery -t sendtargets -p $targe
    iscsiadm -m node -p $targetip -T $targetname --op update \
        -n node.transport_name -v iser
    iscsiadm -m node -p $targetip -T $targetname --login

Related usefull SF question: Set up simple Infiniband Block Storage (SRP or iSER).

Community
  • 1
Veniamin
  • 863
  • 6
  • 11
  • I have changed the transport mode to 'iser' but do not seem to notice much difference, however there may be a few added MB/s which I'm just not noticing due to the inconsistent results. From some further testing, it almost seems like iSCSI is ignoring my MTU of 65520 (for Infiniband). – jaseeey Dec 25 '13 at 14:34
  • I checked out the article you posted... I'm thinking I might need to re-implement this using SRP to get better performance instead of IPoIB. Thank you for your advice! :) – jaseeey Dec 25 '13 at 15:07