Generate List of Valid IP Addresses & Randomly Use in Python Selenium Loop

Question

Disclaimer: This is my first foray into web scraping

I have a list of ~400 search results URLs that I am trying to loop through using Selenium to collect information. At a certain point, I am redirected and presented with the following text:

"Your access to VINELink.com has been declined due to higher than normal utilization levels... You are attempting to access this website from the following ip address. Please make sure your firewall settings are not restricting access. [MY IP ADDRESS]"

Is there a way to generate a list of valid random IP addresses, select one randomly within a loop and feed it to the Selenium WebDriver to avoid being blocked?

I understand that there are ethical considerations to this question (in reality, I've contacted the site to explain my benign use case and ask if they can unblock my real IP address); I'm mostly just interested if this is something one could do.

Abbreviated list of URLs:

['http://www.vinelink.com/vinelink/servlet/SubjectSearch?siteID=34003&agency=33&offenderID=2662',
 'http://www.vinelink.com/vinelink/servlet/SubjectSearch?siteID=34003&agency=33&offenderID=A21069',
 'http://www.vinelink.com/vinelink/servlet/SubjectSearch?siteID=34003&agency=33&offenderID=B59293',
 ...]

Abbreviated code for loop (missing the actual list of valid IP addresses):

info = {}

for url in detail_urls:

    proxy = ### SELECT RANDOM IP ADDRESS FROM A LIST OF VALID IP ADDRESSES ###

    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument('--proxy-server='+str(proxy))
    driver = webdriver.Chrome(executable_path='/PATH/chromedriver', options=chrome_options)
    driver.get(url)
    driver.implicitly_wait(3)

    if drive.find_element_by_xpath('//*[@id="ngVewDiv"]/div/div/div/div[3]/div[3]/div[2]/div/search-result/div/div[4]/div[1]/more-info/div[1]/button'):
        button = driver.find_element_by_xpath('//*[@id="ngVewDiv"]/div/div/div/div[3]/div[3]/div[2]/div/search-result/div/div[4]/div[1]/more-info/div[1]/button').click() 
        name = driver.find_element_by_xpath('//*[@id="ngVewDiv"]/div/div/div/div[3]/div[3]/div[2]/div/search-result/div/div[1]/div/div[1]/span[1]/span[1]/div/div/div[2]/span')
        name = name.text
        offenderid = driver.find_element_by_xpath('//*[@id="ngVewDiv"]/div/div/div/div[3]/div[3]/div[2]/div/search-result/div/div[4]/div[1]/more-info/div[2]/div/div/div[2]/div[1]/div/div[2]/span')
        offenderid = offenderid.text
        info[name] = [offenderid]
        driver.close()
    else:
        driver.close()

Corey Goldberg · Accepted Answer · 2018-11-23T23:39:48.923

0

Is there a way to generate a list of valid random IP addresses, select one randomly within a loop and feed it to the Selenium WebDriver to avoid being blocked?

To get a random item from a sequence, use random.choice(seq) from the random module.

see: https://docs.python.org/3/library/random.html#random.choice

example:

import random

proxies = ['10.0.1.1', '10.0.1.2', '10.0.1.3']
proxy = random.choice(proxies)

Note: Your question sort of doesn't make sense, because you stated that you want to generate list of valid IP addresses. You can't just generate random IP's and expect them to work... you must actually provide the valid IP's to your script. You will need the server infrastructure that provides this (i.e. a pool of working proxy servers bound to each address in your list) because requests will then be routed through these servers. If you are just trying to spoof your IP and don't have a pool of servers to proxy through, the answer is "No, that won't work."

edited Nov 23 '18 at 23:39

answered Nov 23 '18 at 22:26

Corey Goldberg

59,062
28
129
143

Thanks Corey, I think this is sort of what I assumed originally -- in researching this topic there seem to be sites from which one can grab open anonymous IP proxy servers (assuming these are what you're mentioning?) through which one can rotate -- do you know anything about these? If so, do you expose your own system to risk by routing your requests through these? – OJT Nov 23 '18 at 23:19
It seems like these things have got to verge on the illegal also no? – OJT Nov 23 '18 at 23:53
There is not much risk to your system when sending http through a proxy (less than visiting an unknown website in your browser). There is slightly more risk for the receiving system since you will have no idea if the intermediary tampered with the payload. Personally, I would feel fine proxying (with encryption) any non-sensitive requests through a reputable proxy hosting service. However, it really depends on what your requests contain and your aversion to risk. But I wouldn't, for example, proxy unencrypted requests with login credentials through an unknown 3rd party. – Corey Goldberg Nov 23 '18 at 23:53
The legality from the sender's standpoint would depend on what you are actually sending and which laws you abide by :) For example, a Chinese citizen using a proxy to route around a national firewall so he can view blocked Western content might be putting himself at risk. For your case, evading IP address limiting by using rotating proxies likely breaks the site's Terms of Service... It would be more egregious if your real IP had already been banned for abuse. Whether that matters to you is a different question :) – Corey Goldberg Nov 24 '18 at 00:06
Thanks for all the info! Suppose I've already shot myself in the foot by contacting the site, but since they seem to be part of a government-funded program with benevolent cause, maybe they'll feel generous and let me pull the (relatively small amount of) data I need. – OJT Nov 24 '18 at 00:14

Generate List of Valid IP Addresses & Randomly Use in Python Selenium Loop

1 Answers1