1

I am using private rotating proxy provided by (https://proxy.webshare.io/proxy/rotating?) in which each request to rotating proxy receives a new IP address. when I am using requests.get('https://httpbin.org/get', headers=headers, proxies=get_proxy()) it returns a new IP each time whenever I make request. but when using

session = requests.Session()
session.headers = headers
session.proxies = get_proxy()

session.get('https://httpbin.org/get')

It returns same IP each time whenever I make request. How does session object behaves different from requests.get() function in case of proxies.

Ritik Sahu Student
  • 13
  • 3

3 Answers3

1

Session uses previously set up variables/values for each subsequent request, like Cookies. If you want to change the proxy for each request in the session, then use Prepared Requests to set it each time or just put it in a function:

def send(session, url):
    return session.get(url, proxy=get_proxy())

sess = requests.Session()
sess.headers = headers
resp = send(sess, 'https://httpbin.org/get')
print(resp.status_code)

But if you're trying to hide your origin IP for scraping or something, you probably don't want to persist cookies, etc. so you shouldn't use sessions.

aneroid
  • 12,983
  • 3
  • 36
  • 66
  • great, btw why we shouldn't persist cookies, does websites don't require cookies. also when I try to make request without session, it doesn't work ,for every request it gives captcha. i am using same headers for session and without session. – Ritik Sahu Student Dec 20 '20 at 10:20
  • There's no point in rotating proxies if you want to maintain cookies / sessions. The website can still identify you as you. So just use a regular session without changing proxies. (Use just one or none at all.) – aneroid Dec 20 '20 at 11:39
  • Welcome to StackOverflow. Read "[What should I do when someone answers my question?](https://stackoverflow.com/help/someone-answers)", and about [Voting](https://stackoverflow.com/help/privileges/vote-up) and [Accepting](https://stackoverflow.com/help/accepted-answer). – aneroid Dec 20 '20 at 11:43
0

The following code works, and it take a proxylistfile.txt file to check every proxy:

from requests import *
import bs4 
import sys 

if len(sys.argv) < 2:
    print('Usage: ./testproxy.py <proxylistfile.txt>')
    sys.exit()

ifco = 'http://ifconfig.co'
PROXIES_FILE = sys.argv[1]
proxy = dict()

with open(PROXIES_FILE) as file:
    for line in file:
        if line[0] == '#' or line == "\n":
            continue
        line_parts = line.replace('\n', '').split(':')
        proxy['http'] = f'{line_parts[0]}://{line_parts[1]}:{line_parts[2]}'
        try:
            i = get(ifco, proxies=proxy, timeout=11)
            print(f"{proxy['http']} - successfull - IP ---> ", end='')
            zu = bs4.BeautifulSoup(i.text, 'html.parser')
            testo = zu.findAll('p', text=True)[0].get_text()
            print(testo)
        except:
            print(f"{proxy['http']} - unsuccessfull")
            pass

It connect ot ifconfig.co site and return its real ip to check if the proxy works. The output will be something like:

http://proxy:port - successfull - IP ---> your.real.ip

the input file format should be like:

http:1.1.1.1:3128
Lews
  • 426
  • 4
  • 9
-3

I finally switch to another rotating proxy provider (https://www.proxyegg.com) and the issue has been resolved now.

Student
  • 1