14

For a current course I am taking, we are using a remote computer to run our code.

I am coding locally on my MacBook and I'm looking for a good way to keep my local code up to date on the cluster.

The way I was doing it was to have a terminal open for running SCP to copy the directory, and another terminal that was SSH-ed into the cluster for making and running my code.

This seems less than optimal for me. Is there a way that I could automate the sending of the files to the cluster when they are modified?

Or am I stuck with the one-line command to move everything?

Baum mit Augen
  • 49,044
  • 25
  • 144
  • 182
Dan McClain
  • 11,780
  • 9
  • 47
  • 67

5 Answers5

18

Your best option, other than a distributed version control, is using rsync over ssh. I keep a couple of machines in sync by doing the following on each one:

rsync -urltv --delete -e ssh /src.dir user@othermachine:/src.dir

You mentioned using a MacBook - rsync is on Mac OS X. As far as I know, it didn't need to be installed extra. And the beauty of rsync is that it looks for modifications and only copies modified files over. It doesn't do merging of simultaneous modifications like a distributed version control system would, but if you're like me where you do some work on your laptop then some work on your desktop, rsync is the best way to send all the changed files (and only the changed files) from one to the other when you switch modes.

Note: the rsync options used here are:

  • -u, --update skip files that are newer on the receiver
  • -r, --recursive recurse into directories
  • -l, --links copy symlinks as symlinks
  • -t, --times preserve modification times
  • -v, --verbose increase verbosity
  • --delete delete extraneous files from dest dirs, acts as --delete-during

lastly, -e is the option that allows you to specify your remote shell, in this case ssh

physincubus
  • 986
  • 2
  • 11
  • 26
Paul Tomblin
  • 179,021
  • 58
  • 319
  • 408
  • Does rsync run on my comp only? I'm just wondering in case I don't have access to rsync on the cluster. – Dan McClain Mar 19 '09 at 12:38
  • No, you need rsync at both ends of the ssh tunnel. This is very common however so I would definitely suck it and see. – Evan Mar 19 '09 at 12:41
  • I'll have to check this when I get home. One more question, does rsync look for modifications? – Dan McClain Mar 19 '09 at 12:44
  • Yes, rsync does incremental backups. – Wadih M. Mar 19 '09 at 12:45
  • its worth it to mention that if you do not want to update old files that you know haven't changed, you should use the `--ignore-existing` flag and discard the `-u` – physincubus Feb 17 '18 at 03:51
  • @physincubus `-u` will only send files that have changed on the local side. `--ignore-existing` will not overwrite files on the remote side that have changed over there, even if they've changed on the local side. Two very different concepts. – Paul Tomblin Feb 17 '18 at 16:02
16

If you can use rsync, that would probably be the best way.

Would using sshfs to (fuse) mount the remote folder(s) be an option? You could either edit the files directly, or use rsync, unison or any folder to folder synchronisation tool then.

Evan
  • 18,183
  • 8
  • 41
  • 48
4

I know that's not the answer you know but you can setup an SVN or CVS server, which would be so much easier.

Otherwise I'd go for rsync.

dr. evil
  • 26,944
  • 33
  • 131
  • 201
  • Using source control for file sharing is 'easy' as you suggest, but it is also 'nasty'. This approach might be especially problematic in a couple of scenarios. If the files are already source-controlled, then using the existing SCM (source control system) will 'muddy' the history. Alternatively, trying to get a secondary SCM working will be very problematic, whether a second instance of the same SCM or a different SCM. I don't think there's a clean approach on an existing SCM even if source branches are employed. Personally, I'd avoid trying to use SCM, but stick with rsync or the like. – Rhubbarb May 30 '19 at 08:03
2

Even better than rsync is the Unison file synchronizer, which will allow you to make changes at either end and will detect and help resolve any conflicts. It works over ssh.

Norman Ramsey
  • 198,648
  • 61
  • 360
  • 533
  • and it must only be installed on one of the two machines? – iconoclast Sep 22 '18 at 19:19
  • from `unison -doc tutorial`: "It is important that the version of Unison installed on the server machine is the same as the version of Unison on the client machine." So it seem that `unison` must be installed on both machines. – Rhubbarb May 30 '19 at 07:53
1

Probably not what you're looking for, but you should have a look at a DSCM like git:

  1. create a repository in the remote server.
  2. clone it to your development machine using ssh.
  3. add / modify code
  4. use commit and push (again, over ssh) to merge the changes.
J.C. Inacio
  • 4,442
  • 2
  • 22
  • 25