7

I'm trying to mirror a large Mongo database between from a production server to a dev environment by stopping Mongo on both servers and then running the command:

rsync --archive --delete --recursive --verbose --compress --rsh "ssh -t -o StrictHostKeyChecking=no -i key.pem" remoteuser@remotehost:/var/lib/mongodb/ /var/lib/mongodb

It runs fine for a few minutes, but then stopped with the error:

receiving incremental file list
./
collection-228--5129329295041693519.wt
inflate returned -3 (0 bytes)
rsync error: error in rsync protocol data stream (code 12) at token.c(557) [receiver=3.1.1]
rsync: [generator] write error: Broken pipe (32)
rsync error: error in socket IO (code 10) at io.c(820) [generator=3.1.1]

Googling the error suggests it's some sort of network connection problem, but I'm able to connect to both servers just fine.

If I re-run the command, it fails at the exact same file with the same error message. What's causing this error and how do I fix it?

Cerin
  • 3,600
  • 19
  • 61
  • 79

4 Answers4

4

It turns out rsync accesses rsync on the remote server, and the versions where not the same between my servers. I was running 3.1.1 on the destination server, but 3.1.0 on the source server, and apparently this was enough to break the download of certain files. I install 3.1.1 on the source server, and afterwards the transfer worked flawlessly.

Cerin
  • 3,600
  • 19
  • 61
  • 79
2

Solution:

As explained by Cerin here, just upgrade your rsync versions to ensure that they are the exact same on both the sending and receiving PCs!

I had this exact same problem: a broken pipe write error when transferring a very large file over rsync's ssh, with rsync's --compress (-z) flag set.

My rsync error was:

rsync: [sender] write error: Broken pipe (32)
rsync error: error in rsync protocol data stream (code 12) at io.c(837) [sender=3.1.0]

rsync --version from my SENDING PC showed rsync version 3.1.0 protocol version 31, whereas from my RECEIVING PC it showed rsync version 3.1.2 protocol version 31. Therefore, I decided to just upgrade my SENDING PC to version 3.1.2 as well. Once I did that, it worked!

How to upgrade your rsync version:

Simply follow these instructions here to install rsync from source: http://www.beginninglinux.com/home/backup/compile-rsync-from-source-on-ubuntu.

In short:

  1. Check your rsync version so you know what you currently have:

     rsync --version

  2. Download your desired source file for the version you want here: https://download.samba.org/pub/rsync/src/.

  3. In your GUI file manager, right-click and extract it.

  4. Build from source, and install:

     ./configure
 make
 sudo checkinstall

  5. Check your rsync version to ensure it was updated:

     rsync --version

  6. Done!

  7. (I haven't tested this, but apparently): To remove it from your system use:

     dpkg -r rsync

Once I did the upgrade to get both systems' rsync versions the same, it worked perfectly!

Related:

  1. https://bugs.launchpad.net/ubuntu/+source/rsync/+bug/1300367
  2. https://unix.stackexchange.com/questions/242898/rsync-keeps-disconnecting-broken-pipe/547861#547861
  3. Google search for "rsync write error broken pipe" - https://www.google.com/search?q=rsync+write+error+broken+pipe&oq=rsync+write+error+&aqs=chrome.0.0j69i57j0l4.2263j0j7&sourceid=chrome&ie=UTF-8&sec_act=d
  4. [my answer] https://askubuntu.com/questions/791002/how-to-prevent-sshfs-mount-freeze-after-changing-connection-after-suspend/942820#942820
Gabriel Staples
  • 225
  • 1
  • 6
  • In my case, making both server and client run rsync 3.1.3 improved how long rsync would run, but it still eventually crashes. – Tony Ruth Oct 25 '19 at 17:51
  • I've had zero problems with my setup since fixing it. For your case though, try this then: remove rsync's `--compress` (or `-z`) option, and instead use only ssh compression: `-e "ssh -C"`. SSH compression will be less efficient than the rsync compression, but may prove to be less buggy in your case, so it may be your only option. If using ssh compression instead of rsync compression still fails, try disabling compression entirely. Report back what does and doesn't work. – Gabriel Staples Oct 25 '19 at 18:23
  • I found that my issue appears to be due to periodic loss of network connection for ~40 seconds (as determined by running a ping to google.com). After the network connection is reestablished, rsync does not continue the file transfer. I am attempting with the options: rsync -r -P -e "ssh -C" now. – Tony Ruth Oct 25 '19 at 19:17
  • @TonyRuth, any luck with your command above? If that doesn't work, consider trying an auto-reconnecting `sshfs` remote mount (see [my answer here](https://askubuntu.com/questions/791002/how-to-prevent-sshfs-mount-freeze-after-changing-connection-after-suspend/942820#942820)), with regular, "local" `rsync` over that remote disk connection, rather than through an `ssh`-based tunneling `rsync` run. – Gabriel Staples May 21 '21 at 18:09
  • I was eventually able to copy everything. I cannot find the exact command I used but I believe it did use the -r -P and -e options. The critical part was ensuring that the files which had already been copied over were not copied a second time. Then rsync was gradually able to work through copying everything even with the periodic interruptions. – Tony Ruth Jun 07 '21 at 23:03
1

Are you trying yo sync data between local and remote servers, remote server looks like aws ec2 you can use below command to sync data

rsync -ravhz "ssh -i /path/to/EC2_KEY.pem" /path/to/local/files/* EC2_USER@EC2_INSTANCE:/path/to/remote/files

Please check from and to servers before syncing as you might sync in wrong direction

If your are trying to sync from ec2 to your local server then check if you have opened proper ports between servers

Try to telnet first and check connectivity between servers, you have to whitelist ip's and ports as some firewalls may block data transfer

Vijay Muddu
  • 436
  • 2
  • 9
0

I came to this question because I had this problem. After spending 5 hours trying to solve this I finally found a solution. For me the only thing that worked was to change the MTU size. I ran this command:

sudo ifconfig enp42s0 mtu 1024

The default size was 1500 and I changed it to 1024. Run ifconfig to determine the name of your network interface. In my case the name was enp42s0 wich is very weierd. You will probably have something like eth0.

The solution sucks cause now when using Google meet I am not able to share my screen. If I change the MTU size back to the default value 1500 then I am able to share my screen when using Google meet. I wonder what else will stop working after changing the value to 1024.

I am using the linux distribution pop-os that is based on ubuntu. I belive it is a network driver issue.

Tono Nam
  • 322
  • 3
  • 17