0

I was wondering what is the fastest way to transfer emails from one server to another, because i'm sure that i can get this done a lot faster than it is now. My servers are on the same location (Hetzner CX22, Nurnberg, Germany), both of them are CentOS 7. I was searching for some answers and have found this instructions. So, i have adapted command to my needs and got this:

rsync -aHAXxv  --ignore-existing --numeric-ids --delete --progress -e "ssh -T -c aes128-gcm@openssh.com -o Compression=no -x" /var/vmail/ hetzner:/var/vmail/

Unfortunately, transfer is still really slow:

domain.com/some_user/cur/1492584781.M591770P662.server.example.com,S=86421,W=87778:2,S
         86,421 100%   41.21MB/s    0:00:00 (xfr#10093, ir-chk=2681/13130)

domain.com/some_user/cur/1492769222.M475900P29879.server.example.com,S=95001,W=96492:2,S
         95,001 100%   86.79kB/s    0:00:01 (xfr#10094, ir-chk=2680/13130)

domain.com/some_user/cur/1493024424.M847920P18806.server.example.com,S=128451,W=130379:2,S
        128,451 100%  114.87kB/s    0:00:01 (xfr#10095, ir-chk=2679/13130)

domain.com/some_user/cur/1493027412.M159268P19546.server.example.com,S=32063,W=32505:2,S
         32,063 100%   15.29MB/s    0:00:00 (xfr#10096, ir-chk=2678/13130)

domain.com/some_user/cur/1493131148.M829448P8955.server.example.com,S=56940,W=57762:2,S
         56,940 100%   52.02kB/s    0:00:01 (xfr#10097, ir-chk=2677/13130)

domain.com/some_user/cur/1493197257.M528301P23068.server.example.com,S=138236,W=140319:2,S
        138,236 100%  134.32kB/s    0:00:01 (xfr#10098, ir-chk=2676/13130)

domain.com/some_user/cur/1493276821.M593583P7366.server.example.com,S=130140,W=132090:2,S
        130,140 100%  133.22kB/s    0:00:00 (xfr#10099, ir-chk=2675/13130)

domain.com/some_user/cur/1493285721.M142702P9316.server.example.com,S=3088289,W=3128422:2,RS
      3,088,289 100%  122.06kB/s    0:00:24 (xfr#10100, ir-chk=2674/13130)

domain.com/some_user/cur/1493364790.M802086P25403.server.example.com,S=140179,W=142349:2,S
        140,179 100%  124.45kB/s    0:00:01 (xfr#10101, ir-chk=2673/13130)

domain.com/some_user/cur/1493381945.M847016P29043.server.example.com,S=14406,W=14552:2,S
         14,406 100%  937.89kB/s    0:00:00 (xfr#10102, ir-chk=2672/13130)

domain.com/some_user/cur/1493794842.M437136P14205.server.example.com,S=168652,W=171295:2,S
        168,652 100%  129.68kB/s    0:00:01 (xfr#10103, ir-chk=2671/13130)

domain.com/some_user/cur/1493795425.M641361P14342.server.example.com,S=165936,W=168644:2,S
        165,936 100%  236.91kB/s    0:00:00 (xfr#10104, ir-chk=2670/13130)

domain.com/some_user/cur/1493891895.M398132P7857.server.example.com,S=59286,W=60651:2,S
         59,286 100%   25.29MB/s    0:00:00 (xfr#10105, ir-chk=2669/13130)

domain.com/some_user/cur/1493898317.M403724P9570.server.example.com,S=114020,W=115559:2,RS
        114,020 100%   96.07kB/s    0:00:01 (xfr#10106, ir-chk=2668/13130)

domain.com/some_user/cur/1493901199.M975233P10350.server.example.com,S=3206,W=3267:2,S
          3,206 100%  208.72kB/s    0:00:00 (xfr#10107, ir-chk=2667/13130)

domain.com/some_user/cur/1493903083.M47030P10800.server.example.com,S=149206,W=151472:2,S
        149,206 100%  148.38kB/s    0:00:00 (xfr#10108, ir-chk=2666/13130)

domain.com/some_user/cur/1494227302.M871679P15272.server.example.com,S=246876,W=250340:2,S
        246,876 100%   91.91kB/s    0:00:02 (xfr#10109, ir-chk=2665/13130)

domain.com/some_user/cur/1494245675.M774844P19605.server.example.com,S=919,W=943:2,S
            919 100%    1.50kB/s    0:00:00 (xfr#10110, ir-chk=2664/13130)

domain.com/some_user/cur/1494247042.M200981P19987.server.example.com,S=1280,W=1325:2,S
          1,280 100%    2.07kB/s    0:00:00 (xfr#10111, ir-chk=2663/13130)

domain.com/some_user/cur/1494327533.M621859P4444.server.example.com,S=513481,W=520181:2,S
        513,481 100%  181.42kB/s    0:00:02 (xfr#10112, ir-chk=2662/13130)

domain.com/some_user/cur/1494332092.M831578P5734.server.example.com,S=228943,W=231932:2,S
        228,943 100%  103.08kB/s    0:00:02 (xfr#10113, ir-chk=2661/13130)

domain.com/some_user/cur/1494409391.M627692P22069.server.example.com,S=96728,W=98490:2,S
         96,728 100%   86.66kB/s    0:00:01 (xfr#10114, ir-chk=2660/13130)

I've also tried with scp and sftp, but results are pretty same. I have 20 GB of email files and tar is not an option because i don't have enough space on my source server to pack files (and then to transfer and unpack it on the destination server).

I've managed to transfer 6GB so far (for 12 hours).

Ljubisa Livac
  • 173
  • 1
  • 7
  • You don't need to create a tar archive on your source server, output to stdout and pipe that over ssh to your destination: `tar cvzPf - /var/vmail/ | ssh root@dest "tar xzvPf -"` – HBruijn Apr 03 '19 at 08:07
  • in case you don't need file, just remove "-f" parameter... so ``tar cvzP /var/vmail/ | ssh root@dest "tar xzvP"``. ;-) – Kamil J Apr 03 '19 at 08:17
  • Thanks guys for your answers, i didn't know for that possibility! @HBruijn please make an answer so i could accept it :) – Ljubisa Livac Apr 03 '19 at 09:36

1 Answers1

0

Maybe it is not possible in your context, but I use imapsync to migrate mailboxes https://github.com/imapsync/imapsync, avoiding to deal with the "how long it will last to copy stuff".

Just route emails to the new mbox, and let imapsync populate it from the old mbox.

Chaoxiang N
  • 1,283
  • 5
  • 11
  • imapsync is great tool but it need to have credentials for imap access (it can be "shared" mailbox with all the stuff in theory). The question seems to me more system level (administrator) then user. – Kamil J Apr 03 '19 at 08:19
  • Good catch, I think you're right. – Chaoxiang N Apr 03 '19 at 08:41
  • @ChaoxiangN thanks for taking time to answer my question, i've hear of imapsync, but never used it before and it's not free so i didn't want to pay for it right now, maybe some other time.. – Ljubisa Livac Apr 03 '19 at 09:38
  • Extracted from [https://github.com/imapsync/imapsync/blob/master/README], _Imapsync is free, open, public but not always gratis software cover by the NOLIMIT Public License_ => According to me you can use it from time to time freely. On regular usage, I would give money to the author. – Chaoxiang N Apr 03 '19 at 11:10
  • Ahaa, thanks for clarification, i was referring to his website where is not so clear that is open source (https://imapsync.lamiral.info/). Anyway, i've tried imapsync in the meanwhile but unfortunately the speed i get is around 100 kb/s, it's even slower than rsync / scp / sftp. I can't believe that email transfer has to be so slow :( What people do in enterprises where they have hundreds of email accounts and TBs of data... – Ljubisa Livac Apr 03 '19 at 19:28