Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [rsync]

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

Rsync website

1595 questions
7
votes
3 answers

How to make rsync of ~2M files from remote server performant for regular backups

We have a large amount of files on a remote server that I'd like to setup regular backups to a local system for extra redundancy. Some details: Remote system is not in my control. I only have SSH/rsync or FTP access Remote system runs rsync 2.6.6…
jimmy0x52
  • 200
  • 8
7
votes
4 answers

Rsync huge dataset of small files 5TB, +M small files

I encountered a situation where an app server misconfig led to creation of around 5TB datasets where each dir contains huge number of small files. We are in the process of transferring the files and change the application but the rsync fails on…
h.safe
  • 131
  • 1
  • 7
7
votes
4 answers

Unable to rsync due to broken pipe

I'm trying to mirror a large Mongo database between from a production server to a dev environment by stopping Mongo on both servers and then running the command: rsync --archive --delete --recursive --verbose --compress --rsh "ssh -t -o…
Cerin
  • 3,600
  • 19
  • 61
  • 79
7
votes
1 answer

rsync output file sizes in human-readable format

when you use rsync -avzh --stats --out-format="%t %f %b" it does the output in bytes the man pages says the following In addition, one or more apostrophes may be specified prior to a numerical escape to indicate that the numerical value should…
Sl33py
  • 155
  • 1
  • 3
  • 9
7
votes
0 answers

How to re-compile iconv on Linux (Ubuntu 14.04 LTS) with pseudo-charset UTF8-MAC?

I was googling around for hours to find a solution for my problem and I couldn't get it to work: I have to rsync a file structure on an ext4 formatted drive to a hfs+ formatted drive. Folders and file names can contain german Umlauts (äöüß) and the…
Achim Lammerts
  • 166
  • 7
7
votes
2 answers

ignore files in use (being written to) when using rsync

I have an SFTP server where clients are constantly uploading large files. On a periodic basis, I want to copy all complete (fully uploaded) files to another machine for processing. I don't want to copy a file that is actively being written to. Is…
braindancer
  • 73
  • 1
  • 3
7
votes
6 answers

Rsync hangs: expand file_list pointer array to N bytes, did move

Rsync is going into "interruptible sleep" mode after transferring some files from a local folder to a NFS folder. The folder I am trying to backup contains more than 180gb of data. This is what rsync outputs before it hangs: [sender] expand…
jithujose
  • 241
  • 1
  • 3
  • 6
7
votes
3 answers

Why is FileZilla SFTP file transfer max capped at 1.3MiB/sec instead of saturating available bandwidth? rsync and WinSCP are even slower

I'm downloading from a server and downloads are maxing out at 1.3MiB/second with FileZilla but I can start concurrent downloads and they will download at 1.3MiB/second also. So why can't I download just one file at faster than 1.3MB/s and get closer…
htfree
  • 483
  • 4
  • 9
  • 21
7
votes
1 answer

How do "correctly" sync millions of files with rsync and inotify?

Assume I have 1 million files, the file list would be something like 20MB. If I add just one file, inotify will tell rsync to transfer the file list and the new file. My network is not good and I might have to limit the bandwidth…
seanlook
  • 559
  • 2
  • 10
  • 18
7
votes
1 answer

rsync termiates after a while unexpectedly

I run a rsync command with nohup in ssh session, but after a while it returns this error: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [generator=3.0.9] rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at…
Arash Mousavi
  • 678
  • 3
  • 8
  • 21
7
votes
3 answers

Rsync filter and ignore some directories

I'd like to rsync a directory that contains a Python 3 app on my server. The directory has files I'm interested in: the files that end with .py And there are the files I'd like to be ignored (these are the files that are inside a __pycache__…
John Smith Optional
  • 502
  • 2
  • 9
  • 18
7
votes
4 answers

Rsync over SSH getting no tty present

I'm using the following command in root's crontab on Debian. rsync -vqrlHEAXogDtzhi --log-file=${LOG} --progress --rsync-path="sudo /usr/bin/rsync" --exclude-from=$CONFIG_DIR/excludes -e "ssh -i /home/backups/.ssh/id_rsa" backups@${HOSTNAME}:/…
Lynn Owens
  • 435
  • 3
  • 7
  • 14
7
votes
3 answers

Disable reverse lookup in rsync daemon

I'm having an issue with some rsync modules being served through rsync --daemon This lines are from my /var/log/rsyncd.log file : 2013/02/08 12:15:28 [13174] name lookup failed for XXX.XXX.XXX.XXX: Name or service not known 2013/02/08 12:15:28…
Jorge V.
  • 292
  • 1
  • 4
  • 12
7
votes
2 answers

rsync protocol incompatibility

When doing an rsync pull from a remote server using the following command: /usr/bin/rsync -av -e ssh --delete --chmod=a+rwx,g+rwx,o-wx --dry-run username@server:/remote/path/ /home/dir/local/path I get the following error: receiving file list ...…
denormalizer
  • 491
  • 2
  • 5
  • 15
7
votes
3 answers

how to make rsync not uncompressing files?

I'm using rsync to copy files between servers, I'm using -z option what compresses data only for the network transfer stage. But I would like to leave compressed files on remote machine and I have not found this option in rsync. Is this possible…
wlk
  • 1,713
  • 3
  • 14
  • 19
1 2 3
…
99 100