32

I have three windows boxes that are connected to a linux box on my network in various ways (one's local to the network, one is a laptop and may or may not be local, and the other is my work desktop). I'm currently making use of git to syncronize most of my stuff between the boxes (using the Linux box as the server) and it works like a champ.

Now, however, I'd also like to add my MP3 collection (12 gigs) to be efficiently synchronized between boxes. I initially thought I could use git for this, but it looks like rsync would be a better choice since I don't really need all the historical stuff. Since all my existing management scripts run under git bash (instead of cygwin), I'd like to be able to use it from git bash (MINGW32 is what it says in the title bar of the window). Can I just download it in Cygwin and copy it over or is there a better way to approach it? The Linux box is running the latest version of Ubuntu Server edition - do I need to do anything to it to enable rsync to work correctly on it?

I realize I'm probably doing this in a slightly harder way than is needed, but I'm also kind of using this as an exercise in improving my commandline skills. Any suggestions?

I can use rsync from my cygwin shell, but not from msysgit. I tried copying rsync.exe from cygwin's bin folder over to msysgit's bin folder, but it still isn't finding it. Is there some other step I need to do to install it there? It's strange that I'm getting an error that it can't find the file, instead of giving me an error that one of the file's dependencies is missing.

Owen Blacker
  • 631
  • 1
  • 8
  • 20
Will Gant
  • 331
  • 1
  • 3
  • 3

6 Answers6

39

My original instructions to get rsync working with Git for Windows were quite a bit outdated, so I've revamped this answer for mid-2023 to cover MSYS2's switch to zstd and the additional dependencies required for rsync. The primary issue is that MSYS2 now compresses most packages (including zstd, rsync, and dependencies) using zstd, but zstd isn't included in the base environment. Because of this we can't get zstd from the MSYS2 repository--but we can get it directly from the zstd Github releases where it is a zip archive rather than zst. The basic idea for getting rsync working that I originally came across in this post is still the same, but due to the above mentioned issues it requires a bit more work. The whole process is still easily doable once we know everything required.

Below are commands to:
  • download the zstd binary to use in extracting further packages
  • download, extract, and place the rsync binary and it's dependencies in the correct bin folder

These steps have been tested on a clean install of Windows 11 with only the latest Git for Windows installed (should work for Windows 10 as well). No other dependencies should be required to complete the steps.

As always, examine what you paste into a terminal and use at your own risk

Step 1: zstd requirement

If you have an environment that doesn't already support extracting archives compressed with zstd (such as with Git for Windows), the easiest way is probably to download the latest zstd Windows binary directly from the ztsd release page: https://github.com/facebook/zstd/releases. If you already have zstd available and in your sourced path then skip to step 2 (and you may need to adjust the zstd binary location in the script).

Copy the below lines into a Git Bash terminal after adjusting below variables for latest source locations, versions, and your preferred download location:

working="/c/temp-downloads"
zstd_url="https://github.com/facebook/zstd/releases/download/v1.5.5/zstd-v1.5.5-win64.zip"
mkdir -p "$working"
curl -L -o "$working/zstd.zip" "$zstd_url"
unzip -j "$working/zstd.zip" "*/zstd.exe" -d "$working"

Step 2: Downloading compatible rsync and dependency binaries, extracting, and placing in proper location

Copy the below lines into an admin Git Bash terminal after adjusting below variables for latest source locations, versions, and your preferred download location. The terminal must be running as admin if the destination bin folder is in a location requiring privileged access.

# If you downloaded zstd above, make sure your working folder is the same as where the zstd binary is located
working="/c/temp-downloads"
# Set the location of the Git usr/bin folder (under normal installations we can simply use /usr/bin because the MSYS2 root is the Git installation folder)
git_usr_bin_folder_location="/usr/bin"
rsync_url="https://repo.msys2.org/msys/x86_64/rsync-3.2.7-2-x86_64.pkg.tar.zst"
libxxhash_url="https://repo.msys2.org/msys/x86_64/libxxhash-0.8.1-1-x86_64.pkg.tar.zst"
mkdir -p "$working"
curl -L -o "$working/rsync.tar.zst" "$rsync_url"
curl -L -o "$working/libxxhash.tar.zst" "$libxxhash_url"
tar -I "$working/zstd.exe" -xvf "$working/rsync.tar.zst" --directory="$working"
tar -I "$working/zstd.exe" -xvf "$working/libxxhash.tar.zst" --directory="$working"
cp -a "$working/usr/bin/." "$git_usr_bin_folder_location"

That should be all you need to get rsync working as everything else necessary should already be included in the base Git for Windows MSYS2 environment. You can now clean up the contents of the working folder. Hopefully at some point zstd will be included in the base environment and the extra step of acquiring it won't be needed, but it is what it is for now.

If you want to also use Pageant authentication with rsync over ssh, the ssh-pageant binary is already installed in Git for Windows and all you need to do is add a small bit to your ~/.bashrc or ~/.bash_profile file (see ssh-pageant usage section and Git for Windows wiki). I've modified it specifically for use with the Git for Windows environment and enhanced it with some detection for a missing socket file:

# ssh-pageant allows use of the PuTTY authentication agent (Pageant)
SSH_PAGEANT="$(command -v ssh-pageant)"
if [ -S "$SSH_AUTH_SOCK" ]; then
  PAGEANT_SOCK=$SSH_AUTH_SOCK
else
  PAGEANT_SOCK=${TEMP:-/tmp}/.ssh-pageant-$USERNAME
fi
if [ -x "$SSH_PAGEANT" ]; then
  eval $("$SSH_PAGEANT" -qra "$PAGEANT_SOCK")
fi
unset SSH_PAGEANT
unset PAGEANT_SOCK
nextgentech
  • 841
  • 9
  • 8
  • 1
    thank you sir, downloading and it works .. who would have known... – pscheit Nov 18 '17 at 07:46
  • Problem. There is no `rsync.exe` only various versions of rsync in`.pkg.tar.xz` formats. That might be OK, except that I cannot install `apt-get install xz-utils` (`sudo apt-get install xz-utils`) because I can neither install `sudo` OR `apt-get` to Git Bash for Windows :-( and thus thus cannot install `xz-utils`. Is there any way around this? Is there a newer link where I can get `rsync.exe`, or some way I can get Git Bash to uncompress this .xz file? – SherylHohman Jul 23 '19 at 14:09
  • Actually, it seems to be more a permissions problem. When I `tar -xf rsync-3.1.3-1-x86_64.pkg.tar.xz`, I get a bunch of messages stating: `Cannot open: Permission denied`, `Cannot mkdir: Permission denied`, `Cannot mkdir: No such file or directory` and `Cannot open: No such file or directory` for various files/directories. *Maybe* Git Bash already knows how to uncompress `xz` files? But cannot b/e of permissions? I cannot use or install `sudo`. to Git Bash. – SherylHohman Jul 23 '19 at 14:17
  • Assuming you downloaded the tar.xz file to your Downloads folder, you can unpack it from Git Bash using: `cd / && tar Jxvf /c/Users/$USR/Downloads/rsync-3.1.3-1-x86_64.pkg.tar.xz` – hansfn Aug 05 '19 at 13:10
  • @SherylHohman you have to manually extract the exe from the package file. There is no "installation" necessary other than just dropping the exe into the `C:\Program Files\Git\usr\bin` folder. – nextgentech Aug 05 '19 at 18:53
  • Thx for updating your answer with unpack instructions, but why both "cd /c/downloaded_location" and "tar [...] -C /c/downloaded_location/ [...]" in the unpack command? – hansfn Aug 06 '19 at 08:31
  • @hansfn you could leave out the `-C` option in this specific scenario since you would already be in that folder. I just left it for clarity. I was going to have the command extract it directly to the git bin folder until I realized that that the shell would need to be run as admin which is a bit outside the scope here. I'm also somewhat assuming anyone wanting to use rsync via git bash on Windows knows how to copy a file to a folder that requires privileged access. – nextgentech Aug 06 '19 at 18:36
  • OK, my point is that using both isn't clearer - to me. I always find it confusing when unneeded options are added. Why not remove the cd command then? I fell slightly in the same trap myself by using "J" to indicate that we are unpacking a xz file - old habits. By the way, if you run Git Bash as administrator (and unpack from the root as I did), you 1) don't have to know where Git for Windows is installed and 2) you don't have to use a "complex" tar command. Anyway, this is nitpicking ;-) – hansfn Aug 07 '19 at 06:18
  • @hansfn I don't think most people would be running Git Bash as administrator, and many commands will silently fail if there are permission issues. I appreciate the desire to clarify the procedures, but basically I think that people trying to do this will have enough experience that we don't need to clutter things up with too many instructions. – nextgentech Aug 07 '19 at 21:58
  • 1
    In my case i had to put rsync.exe in ```c:\Program Files\Git\mingw64\bin\``` – kanlukasz Feb 15 '20 at 14:31
  • 8
    An update: since 2020, there's one more step required first. The rsync packages in that repo are no longer in `.tar.xz` format but rather `.tar.zst`, and Git for Windows out of the box doesn't understand that format. Happily, it's enough to first install `zstd`: download `zstd-….pkg.tar.xz` from that same repository, unpack and install that, and then `tar -xvf` is able to handle the `rsync.pkg.tar.zst` file. – Greg Price Mar 16 '21 at 21:19
  • 5
    Further to Greg Price's info relating to zstd now being required: you cannot pick the latest zstd package because it is bundled in zstd format (chicken and egg, much?) but there is an older version with .xv. Also my experience was that calling rsync having put zstd and rsync in /usr/bin results in rsync --version complaining that msys-zstd-1.dll is not found, so unpacking-the-zstd-package-into-\usr\bin instruction is incomplete – Jeremy Taylor Aug 02 '21 at 22:08
  • 4
    I found it was necessary to d/l rsync, libxxhas, liblzr, and libzstd, unpack with 7-Zip-zstd, and copy into /usr/bin – Jeremy Taylor Aug 02 '21 at 22:40
  • you can also unpack in `wsl` but you probably need to `sudo apt-get install zstd` first – CervEd Oct 10 '21 at 11:49
  • @JeremyTaylor `liblzr` doesn't exists in http://repo.msys2.org/msys/x86_64/ has it moved or did you find it somewhere else? – CervEd Oct 10 '21 at 12:07
  • 2
    I was able to get `rsync` working with only `libxxhas` and `libsztd`. But I did the untaring in `wsl` – CervEd Oct 10 '21 at 13:36
  • To confirm: on Git for Windows 2.36 using git bash, I had to download 1. rsync, 2. libxxhash and 3. libzstd from the msys repo. I installed `zstd` on my linux machine. I extracted the zst files using `tar --use-compress-program=unzstd -xvf -C `. The resulting exe and dll files I copied into `C:\Program Files\Git\usr\bin`. Open a new Git Bash window. Bob's your uncle. – klaar Apr 26 '22 at 15:05
  • To confirm on the date of this comment: Rsync-3.2.3-1 worked with libxxhash-0.8.0-1, not sure about libzstd, but I have it installed too, with 1.5.2-2. Cost me an entire afternoon. – The-Vinh VO Jan 12 '23 at 10:24
7

I know the question isn't "Using rsync for MSYS", but I think mentioning this is relevant:

"MinGW Shell" (formerly best known as "MSYS (Basic System)") does indeed have rsync. I'm using that shell/system as often as possible since it's much smaller than Cygwin. (I'm probably the only one that cares about file sizes these days.)

"Git for Windows" / "Git Bash" could easily add rsync since it based on MinGW/MSYS and it has been mentioned on the mailing list earlier.

PS! If your MinGW Shell doesn't have rsync, just run "mingw-get install msys-rsync" to add it.

Updated 25th of March 2016:

Git for Windows now uses MSYS2. And MSYS2 uses "pacman", not "mingw-get", to install packages. Unfortunately, "Git for Windows" doesn't include pacman ... There are two options:

  1. Install the Git for Windows SDK (which includes pacman).
  2. Install Git inside MSYS2.

The last option is my favorite, but it's not streamlined yet so the first option is probably easier. Hopefully, it will be fixed.

PS! I haven't played much with MSYS2 / pacman, but it seems very promising.

Updated 7th of August 2019:

If you are using Git for Windows (Git Bash) it's much better to just grab the rsync package from the MSYS2 Package repository, as is described in the answer above.

hansfn
  • 195
  • 1
  • 7
  • sadly, in Git Bash: `$ mingw-get bash: mingw-get: command not found` – Hendy Irawan Mar 24 '16 at 17:06
  • Yes, it's a bit sad. Look at my updated answer above - still sad, but the situation has improved. – hansfn Mar 25 '16 at 08:40
  • I went for the second option and I ended up with a 2.17GB folder, but everything work like a charm. I installed rsync by using `pacman -S rsync`. In the past I used CygWin but I think MSYS2 is awesome, I don't have to run a gui to install extra packages. – Edenshaw Jun 21 '16 at 16:39
  • The bug report listed in this answer has since been closed as "unfixable" - however, I have created a [pull request to have rsync included with Git for Windows](https://github.com/git-for-windows/build-extra/pull/131). – Scott Stevens Feb 14 '17 at 12:54
  • Any links on how to do either option 1 or option 2? I think I have Git for Windows SDK. Dunno. But assuming I do, or can get it.. what then? If not, I have no idea what it even means to install Git inside MSYS2. Does that mean I need to delete my current Git install. Will I loose anything, like settings or functionality if I do so. Will nay links break? – SherylHohman Jul 23 '19 at 14:22
  • If you have installed Git for Windows - you probably don't have the SDK if you didn't choose it explicitly - just follow the advice in the top voted answer (above): https://serverfault.com/a/872557/94895 – hansfn Aug 05 '19 at 12:56
2

Update January 2021:

Since mysys2 is now using zstd you need to download more than just rsync to get this working. As well as rsync-3.2.3-1-x86_64.pkg you also need to download and copy libzstd-1.4.8-1-x86_64.pkg libxxhash-0.8.0-1-x86_64.pkg and for completeness I also downloaded zstd-1.4.8-1-x86_64.pkg

and 7-zip doesn't do the job of extracting the .zst compressed files wither, so I had to get PeaZip.

PhilipSargent
  • 21
  • 1
  • ..except that it seems to be unusably slow! – PhilipSargent Jan 03 '21 at 17:40
  • Aha. I have a network drive mounted as P: so I can rsync to it either as /p/mycopyfolder/ or 192.168.0.14::/mycopyfolder/ The /P/ method is unusably slow whereas explicitly using the IP address is as fast as I would expect. Interestingly I get exactly the same effect when using cygwin64 instead of mysys2 : the IP address version is fast but /cygdrive/p/ is unusably slow. msys2: rsync version 3.2.4dev protocol version 31 cygwin64: rsync version 3.2.3 protocol version 31 – PhilipSargent Jan 03 '21 at 23:37
  • Thank you for this 2021 update! To unpack the .zst files you could also install [7-Zip Zstandard Edition](https://awesomeopensource.com/project/mcmilk/7-Zip-zstd) And [this link](https://tlundberg.com/blog/2020-06-15/installing-rsync-on-windows/) explains how you can run rsync directly from a command prompt window. – FlexMcMurphy Mar 21 '21 at 03:41
2

I assume by git bash you really mean msysgit, aka "git for windows." The Start Menu option is git bash, which explains the name you used..

First, you are correct in dismissing git as an option here. One of the things git does not do well is handling binary files--and you'd end up with a huge, bloated repository. Additionally, since you won't be handling changes, using the overhead of git strikes me as overkill.

Secondly, it sounds like the question you're actually asking is, "Can I use rsync from msysgit/Windows?" The short answer is, not out of the box. msysgit is really just the Windows command line in terms of form and function. That being said, you have two options: download a third-party tool to enable rsync on your Windows machine (Stack Overflow: Rsync for Windows), or just use cygwin. Really, they have the same outcome--so its up to you to decide which ones work better.

Community
  • 1
Andrew M.
  • 11,182
  • 2
  • 35
  • 29
  • I can use rsync from my cygwin shell, but not from msysgit. I tried copying rsync.exe from cygwin's bin folder over to msysgit's bin folder, but it still isn't finding it. Is there some other step I need to do to install it there? It's strange that I'm getting an error that it can't find the file, instead of giving me an error that one of the file's dependencies is missing. Thanks, Will – Will Gant Sep 13 '11 at 01:31
  • This simply isn't possible. cygwin sets up a GNU-based environment, including Linux-like libraries, configs, etc.. `msysgit` is the Windows command line, which supports Windows executables. What's likely happening is, its not even able to detect the file as executable. Simply put, cygwin and msysgit are not even remotely compatible. – Andrew M. Sep 13 '11 at 02:08
  • It is possibly but hacky. Copy `cygiconv-2.dll`, `cygwin1.dll` and `rsync.exe` from Cygwin to your msysgit path. Nothing else (especially not ssh.exe). The problem is usually an incompatibility between Git SSH and Cygwin/rsync SSH. Try to keep the same version. You'll still have issues with the `/cygdrive/` but there I've no solution except a `cd`. – Wernight Jun 18 '12 at 23:47
0

rsync is a great for unidirectional sync. You can either run rsync under cygwin, or for Windows there is also Deltacopy

If you wanted to have bidirectional sync, then take a look at unison

For a cross platform directory synchronisation tool, there is syncthing which runs on most platforms.

shouldbeq931
  • 509
  • 4
  • 15
  • unison is great, but you have to install it on server - like 90% of servers are rsync ready. And OP asked for rsync – pscheit Nov 18 '17 at 07:50
  • the OP asked about rsync, but the what the OP was trying to achieve was "to be efficiently synchronized between boxes", and unison is available in most repos... – shouldbeq931 Nov 19 '17 at 11:20
  • Looks like DeltaCopy is not available for Windows 10. May no longer be maintained? – SherylHohman Jul 23 '19 at 17:48
  • It's difficult to tell if Unison works on Windows 10. Hasn't had an update in a long while. There is no mention of what versions of Windows OS that this software supports. There are several issues re: Windows 10 https://github.com/bcpierce00/unison/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+Windows+10 – SherylHohman Jul 23 '19 at 17:54
  • As a cross platform tool to keep a directory structure in sync these days, I'd suggest syncthing https://syncthing.net/ However in 2011.... – shouldbeq931 Jul 24 '19 at 20:19
0

For what it's worth, I was having similar problems getting cwRsync to work in msysgit when dropped into the bin path, but after trying lots of different things eventually found that simply updating to the latest version (2.10.2) of Git for Windows fixed the issue.

Nick F
  • 111
  • 2