0

I have use case where I need to move some payload from one server to the other. Both servers are placed within the same data center (even in the same Amazon's availability zone).

I have service / endpoint, that needs to trigger a file transfer whenever called through HTTP/HTTPS. I need to speed up the transfer, as it currently can even take 15+ seconds. Making it asynchronous (eg. handled with task queue like Celery) not necessarily solves my problem (transfer still needs to be performed quickly - does not matter much, whether during the request that triggered it, or outside of it).

The files are generally like 100KB-500KB in size and I have the following options:

  • SSH transfer,
  • other secure transfers between file systems (SSHFS?, rsync through SSH?),
  • HTTP/HTTPS request with file as a body,
  • something else?

The files are always new (so sending a diff between previous version and newer one is not applicable). They are text files, so compression could possibly speed up the transfer.

Does anyone have thoughts on the fastest way to do that (quickly and securely transfer file between servers)? Servers are separate instances and most likely will stay this way. Also security plays significant role. But outside of these restrictions, I am able to install stuff on the servers or mount directories.

Tadeck
  • 132,510
  • 28
  • 152
  • 198
  • You're saying that a single 100KB-500KB file is taking >15+ seconds to be transferred from one server to another within the same data center? That seems seriously wrong. – ruakh Nov 18 '13 at 03:44
  • @ruakh: Well, the bigger tasks can take that order of magnitude to transfer. And that 15+ seconds time frame includes starting SSH library and establishing a connection. But I agree, that there is something wrong. I am trying to clarify that. This is also why I want to know the fastest option. – Tadeck Nov 18 '13 at 04:01
  • Are you completely replacing the files, or do some stay the same ? – Mohammad AbuShady Nov 18 '13 at 06:41
  • @MohammadAbuShady: These act as messages, so files can be copied and then can totally disappear, but can also be stored for later reference. So, answering your question: there will be no replacing of the files, they will all be new (in terms of file name). – Tadeck Nov 18 '13 at 12:14

1 Answers1

-1

Well the answer I had in mind is as following

  • First server would compress all the new messages that needs to be sent ( gzip or whatever )
  • The compressed file get moved between the two servers ( SCP should be fine, since it's only 1 file )
  • The second server would uncompress the files in the messages folder

I thought of the compressing first to increase the file size instead of the files count, this way you would reduce the communication time between the two servers in cost of some processing time, which I assume should be less than the time of recursively moving all those files.

Mohammad AbuShady
  • 40,884
  • 11
  • 78
  • 89