0

I transfer from 1 linux host to another linux host around 15TB data. The sender host have those network cards:

[44.206701] igb: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

[44.274112] bonding: bond0: link status definitely up for interface eth0, 1000 Mbps full duplex.

[44.278534] igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

[44.573636] bonding: bond0: link status definitely up for interface eth1, 1000 Mbps full duplex.

The receiver host have this network card:

[16.964486] e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None

My max transfer speed right now is 11 MB/s. I'm wondering, what do I need to do to increase the transfer speed? upgrade the receiver network card?

Thanks!

Shlomi
  • 331
  • 2
  • 9
  • 19
  • 3
    `e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex` - That is a 1Gbps nic, but it is linking at 100Megabits per second. Figure out why it isn't linking at 1Gbps. Oh, and 100Megabits/8=~12MB/s. So right now you are getting the maximum speed. – Zoredache Mar 16 '17 at 22:01
  • 99% of the time a 1000BASE-T (1 Gbps) NIC connects at 100BASE-TX (100 Mbps) is because the cable is bad. 1000BASE-T requires all four pairs of wires be correctly configured in the cable. If even one wire doesn't work, the auto-negotiation will step down the connection to 100BASE-TX. – Ron Maupin Mar 17 '17 at 04:22

5 Answers5

6

The receiver side is linking at only 100 Mb/s:

e1000e: eth0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: None

which translate to about 12.5 MB/s of maximum raw bandwidth. After taking into account the IP/TCP overhead, at 11 MB/s you are basically maxing your receiver bandwidth.

shodanshok
  • 47,711
  • 7
  • 111
  • 180
0

it would better to have a 1gig network card on the receiving machine yes, also, i can see in the tags that you are using scp.

scp is usually very bad for important data transfers, i always had horrible performance with it

try ftp (but it's unencrypted), you will clearly notice an important difference with the transfer speed.

ftp (or another protocol/program but not scp) + at least a 1000mb NIC on receiving machine will clearly help you

olivierg
  • 524
  • 2
  • 8
  • 27
  • Hi, I'm actually using rsync, I dunno why I tagged 'scp' :P – Shlomi Mar 16 '17 at 21:58
  • thanks for your answer, rsync uses ssh (which is the same protocol as scp), try using ftp just to compare, change your NIC on the receiving box, and be careful with disk I/Os too, as billthor said ! – olivierg Mar 16 '17 at 22:12
0

As others have suggested, the reason why you are only getting 11Mb/sec is because the receiving network card has a link speed of 100MBit, despite the card theoretically being able to handle 1GBit.

The only possible reasons I can think of is that either the NIC is connected to a 100MBit switch, or the card is configured to auto-negotiate speed and has negotiated the wrong speed. In that case you should be able to force the NIC to run at 1GBit.

At the current transfer rate it will take you about 15 days to transfer 15Tb (http://www.thecloudcalculator.com/calculators/file-transfer.html).

With a 1GBit connection on the receiver side it will only take about 1.5 days.

Lucky Luke
  • 1,634
  • 1
  • 11
  • 12
  • If autonegotiations doesn't work it must be disabled on both physical interfaces: switch and server. Speed must be configured the same on both interfaces. – Mikhail Khirgiy Mar 17 '17 at 05:49
  • @MikhailKhirgiy *Never* disable autonegotiation on 1GBit or higher-speed links. Doing so places the link outside the IEEE specifications. [The standards for 1000BASE-T, 1000BASE-TX and 10GBASE-T require autonegotiation to be always present and enabled. Other than speed and duplex mode, autonegotiation is used to communicate the port type (single port or multiport) and the master-slave parameters (whether it is manually configured or not, whether the device is master or slave if this is the case, and the master-slave seed bit otherwise).](https://en.wikipedia.org/wiki/Autonegotiation#Overview) – Andrew Henle Mar 17 '17 at 12:57
  • Disabling autonegotiation to force gigabit speed when it otherwise doesn't work is like getting a flat on your car, removing the wheel, tossing it in the ditch, and continuing to drive. When your car gets a flat tire, *you fix the flat tire*. – Andrew Henle Mar 17 '17 at 13:00
  • 1
    I wrote if you decide to disable autonegotiations then you need to disable it on both ports. I didn't say that disabling autonegotiations is a good idea. But sometimes it helps when equipments​ aren't compatible. – Mikhail Khirgiy Mar 17 '17 at 14:37
0

both sides should be at 1000full if you have a bad cable , forced/*hardcoded the speed on the switch side it will sync at 100 Full on the receiving end.

look at how the switch is configured if its auto auto put your interface hardcoded at 1000full if it does not go up at that speed check cabling. up to switch.

at 12MB you are at 100Full speed . e1000e is a gig interface should sync up to that speed . your transfer should go from 300 to 800 Mbps (30MB to 80MB) depending on the host capabilities.

bob
  • 142
  • 2
-1

You are transferring data close to the limit of the receiving interface, although it not running at its maximum speed. Can your switches handle the speed?

Your bottleneck may be the speed at which you can read from or write to disk or it may be bottleneck in the network. A few questions:

  • How are you transferring the data? (Disk images might be faster than individual files.)
  • What kinds of disk do you have on the systems? (Flash drives are generally faster than hard disks. Raid systems can add write time overhead. Interface speeds are often higher than read/write speeds.)
  • What is their transfer rates? (It doesn't matter how fast the network is if you can't read or write the data at near the network transfer rate.)
  • How large are the files you are transferring? (Small files have more file system overhead.)
  • Does the target file system use a journal? (This adds additional write overhead unless you disable the journal.)
  • Are you buffering data being transferred over the network? (Buffering allows data that can not yet be written to be transferred over the network.)
  • Can you run multiple transfers in parallel? (If the transfers run on different disks, you may gain on overall transfer rate. Too many parallel transfers will add additional overhead that can slow down file movement.)
  • Is the transfer stream encrypted/compressed? (Either can add overhead. Compression can compensate for a network bottleneck.)
  • Are there any errors on any of the interfaces? (Duplex mismatches can cause issues.)
  • Are the network cards/switch ports running at full capacity/speed? (If any device on the network is running at full capacity, it becomes a bottleneck. If they are not running at full speed, it lowers the available throughput.)

As noted in my comment, the receiving system's Ethernet card appears to be running at 1/10th is maximum speed. The transfer rate you are achieving is close to the maximum achievable without compression at the current network card speed.

Running rsync in daemon mode on one side may provide better performance than tunneling using SSH. There are security implications to running rsync as a daemon.

BillThor
  • 27,737
  • 3
  • 37
  • 69
  • 1. I'm using rsync. 2. The systems have WD Blue 2TB disks - The sender host is with 20 disks Raid 10. 3. they are SATA 6.0Gb/s. 4. some files are 500 GB+ some are 100GB, I have total of 15 TB. 5. The root system is currently in 'Read-Only' mode because the os hard disk failed and have corrupted files which fsck cannot fix. – Shlomi Mar 16 '17 at 22:30
  • The WD Blue disk should be able to keep up with the network for such large files. Your transfer rate is about 2 times the sender disk speed. Rsync overhead could account for the difference between network capacity and transfer throughput. I would look at why you are getting 100Mbps rather than 1000Mbps on the network card. This is likely either the router/switch or a bad cable. It may be possible to switch either without restarting the network. – BillThor Mar 17 '17 at 02:11
  • @Billthor at only 11 MB/s, nor the filesystem nor the disks can be the bottleneck. – shodanshok Mar 17 '17 at 06:40
  • @shodanshok Disk and filesystem should not be an issue on a modern system at that rate, But Iv occasionally work on older but still active systems where these speeds can be bottlenecks. As noted in my comment, the network card on the receiving system is not running at full speed and may be the bottleneck. – BillThor Mar 17 '17 at 22:52