How to distribute torrent files?

Question

I am doing some distributed work with RackSpace cloud servers and I am using bittorrent to distribute my files. It works surprisingly well. However, distributing the torrent files themselves are not so nice. How would you go around doing that? Right now I just scp the torrent files to the servers, and of course I could write a script that copies it to sqrt(n) servers instructing each to again copy to sqrt(n) but that's a pita to work it.

chutz · Accepted Answer · 2012-10-31T17:12:44.820

23

Not knowing what exactly your problem is, I can recommend pscp from parallel-ssh as a tool to upload small files to multiple servers.

You prepare a list of servers to upload to and let it know what to take locally and where to put it remotely. For example:

$ pscp -h list-of-servers file.torrent /tmp/
[1] 02:11:22 [SUCCESS] 10.0.0.21
[2] 02:11:22 [SUCCESS] 10.0.0.20
[3] 02:11:22 [SUCCESS] 10.0.0.45
[4] 02:11:22 [SUCCESS] 10.0.0.19
[5] 02:11:22 [SUCCESS] 10.0.0.2
[6] 02:11:22 [SUCCESS] 10.0.0.5
[7] 02:11:25 [FAILURE] 10.0.0.3 Exited with error code 1

edited Oct 31 '12 at 17:12

answered Oct 31 '12 at 17:05

chutz

7,888
1
29
59

1

There's also http://code.google.com/p/pdsh/ – chx Oct 31 '12 at 20:03

score 15 · Answer 2 · answered Oct 31 '12 at 16:56

15

Given they're very small files typically couldn't you just have a 'dropbox' using HTTP with a cron'ed script doing a curl/wget with wildcard?

answered Oct 31 '12 at 16:56

Chopper3

101,299
9
108
239

2

This is the method I would use, no need to push a file out to each server and maintain a server list, have them periodically check for new .torrent file – jwbensley Oct 31 '12 at 17:41
2

you could also use Puppet to do this – Noah Yetter Nov 01 '12 at 05:20

chewbakka · Answer 3 · 2012-11-02T12:58:31.970

8

You could have the torrent clients check an rss feed for new torrent files or switch to magnet links instead of torrent files.

edited Nov 02 '12 at 12:58

answered Oct 31 '12 at 17:36

chewbakka

401
4
8

RSS may not be an option if he/she is using a basic command line torrent client. – Hengjie Oct 31 '12 at 22:44
He could you [Flexget](http://flexget.com/) with rtorrent without gui – chewbakka Nov 01 '12 at 11:07
Yep, flexget and rtorrent would work well for this – Grant Nov 02 '12 at 13:09
+1 "switch to magnet links instead of torrent files." – michael Jan 11 '14 at 12:14

score 2 · Answer 4 · answered Nov 03 '12 at 13:38

You might want to look into murder

Murder is a method of using Bittorrent to distribute files to a large amount of servers within a production environment. This allows for scaleable and fast deploys in environments of hundreds to tens of thousands of servers where centralized distribution systems wouldn't otherwise function

.

Murder was/is developed by the folks at twitter, and they're using it daily to distribute files.

project is no longer maintained. – chovy Aug 30 '18 at 09:02 — chovy, Aug 30 '18 at 09:02

How to distribute torrent files?

4 Answers4