-1


I would like to know if it is possible for a Ruby program to possess multiple IP addresses? I am trying to download a lot of data from a site, but it is very slow with only 1 connection at a time.
I intend to multi-thread my program with each thread using its own IP address, but I do not know if it is possible in the first place, any help or hints would be greatly appreciated.

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Matthieu Raynaud de Fitte
  • 2,001
  • 3
  • 25
  • 49
  • 1 ip per machine, the only way to effectively do so would be have multiple machines, connect to a proxy or vpn and spawn your process through that. – johnny 5 Jun 19 '16 at 01:34
  • I cannot have more than one machine so I need to use multiple IP addresses. Is it possible to use vpn on a thread of a program ? – Matthieu Raynaud de Fitte Jun 19 '16 at 02:09
  • It might be better if you broke out your program into a few components, 1 master which will connect to a vpn programmatically, 2 the component that scrapes the data. You'll need to look for a vpn with an api todo so. Also you might want to test the connection to figure out the rate limit on the website, and you should use a random variable to modify the call time because some website are looking for succinct calls. Alternatively you can try to write your scraper using Google docs depending on the data. – johnny 5 Jun 19 '16 at 03:20
  • There's lots of ways to download through a proxy in ruby. I like Mechanize personally. – pguardiario Jun 19 '16 at 07:22
  • thanks for the advice – Matthieu Raynaud de Fitte Jun 19 '16 at 11:11

1 Answers1

1

It is definitely possible for a machine or a program to have multiple IP addresses. You can even have multiple network adapters, and tie each of them to different physical connections.

However, it can get really hairy to maintain. The challenge for that is partly in the code, partly in the system maintenance, and partly in the networking required to make that happen.

A better approach that you can take is to design your program so that it can run distributed. As such, you can have several copies of it synchronized and doing the work in parallel. You can then scale it horizontally (build more copies) as required, and over different machines and connections if required.

EDIT: You mentioned that you cannot scale horizontally, and that you prefer to use multiple connections from the same machine.

It's very likely that for this you'll have to go a little bit lower in the network stack, developing yourself the connection through sockets in order to use specific network interfaces.

Check out an introduction to Ruby sockets.

Also, check out these related questions:

Community
  • 1
  • 1
Alpha
  • 7,586
  • 8
  • 59
  • 92
  • I am using multiple machines but I run into other issues : they do not have the same specs at all ( do not have the money to invest in any more either ). Therefor, 1 program with multiple IP addresses is the best available solution. any clue on how to approach the problem ? – Matthieu Raynaud de Fitte Jun 19 '16 at 02:06
  • If it's the connection that's slow, you need to add more connections. I'm talking about independent network connections, not just network adapters. I don't have any more info to add right now, sorry. I'll see later if there's a gem or something that can be used to have control of http connections at a low level. – Alpha Jun 19 '16 at 02:09
  • @MatthieuRaynauddeFitte I have updated my answer. Hopefully this will help! – Alpha Jun 19 '16 at 21:45