1

I have a situation where I have a series of mostly connected nodes that need to sync a pooled dataset. They are files from 200-1500K and update at irregular intervals between 30min to 6hours depending on the environment. Right now the numbers of nodes are in the hundreds, but ideally, that will grow.

Currently, I am using libtorrent right now to keep a series of files in sync between a cluster of nodes. I do a dump every few hours and create a new torrent based on the prior one. I then associate it using the strategy of BEP 38. The infohash is then posted to a known entry in the DHT where the other nodes poll to pick it up.

I am wondering if there is a better way to do this. The reason I like BitTorrent was originally for firmware updates. I do not need to worry about nodes less than awesome connectivity, and with the DHT it can self assemble reasonably well. It was then extended to sync these pooled files.

I am currently trying to see if I can make an extension that would allow me to have each node do an announce_peer for each new record. Then in theory interested parties would be able to listen for that. That brings up two big issues:

  1. How do I let the interested nodes know that there is new data?
  2. If I have a thousand or more nodes adding new infohashes every few minutes what will that do to the DHT?

I will admit it feels like I am trying to drive a square peg into a round hole, but I really would like to keep as few protocols in play at a time.

Ian Reinhart Geiser
  • 454
  • 3
  • 5

1 Answers1

0

How do I let the interested nodes know that there is new data?

You can use BEP46 to notify clients of the most recent version of a torrent.

If I have a thousand or more nodes adding new infohashes every few minutes what will that do to the DHT?

It's hard to give a general answer here. Is each node adding a distinct dataset? Or are those thousands of nodes going to participate in the same pooled data and thus more or less share one infohash? The latter should be fairly efficient since not all of them even need to announce themselves, they could just do a read-only lookup, try to connect to the swarm and only do an announce when there are not enough reachable peers. This would be similar to the put optimiation for mutable items

the8472
  • 40,999
  • 5
  • 70
  • 122