1

I want to rotate IP addresses for web-scraping, and here is my setup:

  1. I have configured multiple IP addresses as below in my /etc/network/interfaces file:

    # The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
allow-hotplug eth0
iface eth0 inet static
    address XX.XXX.XXX.146
    netmask 255.255.255.248
    network XX.XXX.XXX.144
    broadcast XX.XXX.XXX.151
    gateway XX.XXX.XXX.145
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 208.67.222.222 208.67.220.220
    dns-search host.myhost.com
auto eth0:1
iface eth0:1 inet static
address XX.XXX.XXX.147
netmask 255.255.255.248
broadcast XX.XXX.XXX.151 
network XX.XXX.XXX.144 

auto eth0:2
iface eth0:2 inet static
address XX.XXX.XXX.148
netmask 255.255.255.248
broadcast XX.XXX.XXX.151 
network XX.XXX.XXX.144 

auto eth0:3
iface eth0:3 inet static
address XX.XXX.XXX.149
netmask 255.255.255.248
broadcast XX.XXX.XXX.151 
network XX.XXX.XXX.144

  2. Cloudflare DNS performs the round-robin to different IP addresses of my server.

  3. However, when I use the following PHP script to check my external IP address, I get different values for $_SERVER['SERVER_ADDR']; but my external IP address remains same (as checked by the script below from http://checkip.dyndns.com/).

    <?php
$externalContent = file_get_contents('http://checkip.dyndns.com/');
preg_match('/Current IP Address: ([\[\]:.[0-9a-fA-F]+)</', $externalContent, $m);
$externalIp = $m[1];
echo $externalIp;
echo '<br/>';
echo $_SERVER['SERVER_ADDR'];
?>

What am I missing here, I want to execute an external executable which should use different public IP addresses available on my server in rotation?

masegaloeh
  • 18,236
  • 10
  • 57
  • 106
iTech
  • 355
  • 4
  • 15

2 Answers2

3

Figured this out, the solution is to configure iptables rules as below:

iptables -t nat -I POSTROUTING -m state --state NEW -p tcp --dport 80 -o eth0 -m statistic --mode nth --every 1 --packet 0 -j SNAT --to-source XX.XXX.XXX.146 iptables -t nat -I POSTROUTING -m state --state NEW -p tcp --dport 80 -o eth0 -m statistic --mode nth --every 2 --packet 0 -j SNAT --to-source XX.XXX.XXX.147 iptables -t nat -I POSTROUTING -m state --state NEW -p tcp --dport 80 -o eth0 -m statistic --mode nth --every 3 --packet 0 -j SNAT --to-source XX.XXX.XXX.148 iptables -t nat -I POSTROUTING -m state --state NEW -p tcp --dport 80 -o eth0 -m statistic --mode nth --every 4 --packet 0 -j SNAT --to-source XX.XXX.XXX.149

iTech
  • 355
  • 4
  • 15
1

If you want to use different source IP addresses when using the file_get_contents PHP function, you should try using PHP contexts and specifying the bindto socket option.

If you're NOT using PHP and your client connection does not provide an option to bind to a specific source address, then you nave no other chance than iptables source-nat: you can define proper iptables nat translations for each of your outgoing IP addresses.

Please note that DNS does NOT nave any role for what you need, as it relates only to INBOUND services.

Damiano Verzulli
  • 4,078
  • 1
  • 21
  • 33