Python stem+requests: Not switching circut/changing IP address when using a session

Question

Cnosider the following script tortest.py, which repeatedly switches the TOR circuit and checks if the IP address has changed:

#!/usr/bin/env python2.7
# -*- coding: utf-8 -*-

import json
import time

# pip install requests[socks]
import requests

# pip install stem
from stem import Signal
from stem.control import Controller

http = requests.session()
proxies = {'http': 'socks5://127.0.0.1:9050', 'https': 'socks5://127.0.0.1:9050'}


def get_new_ip():
    with Controller.from_port(port=9051) as controller:
        controller.authenticate(password="xxx")
        controller.signal(Signal.NEWNYM)
        time.sleep(controller.get_newnym_wait())


for _ in xrange(5):
    get_new_ip()
    try:
        ip1 = json.loads(requests.get("https://httpbin.org/ip", timeout=(5, 5), proxies=proxies).content)["origin"]
        ip2 = json.loads(http.get("https://httpbin.org/ip", timeout=(5, 5), proxies=proxies).content)["origin"]
        print "Without session: {}".format(ip1)
        print "With session:    {}".format(ip2)
        print
    except Exception as e:
        pass

Running this script, I get the following output:

$ python2 tortest.py
Without session: 137.74.169.241
With session:    137.74.169.241

Without session: 145.249.104.203
With session:    137.74.169.241

Without session: 5.189.170.221
With session:    137.74.169.241

Without session: 192.36.27.6
With session:    137.74.169.241

Without session: 199.249.224.43
With session:    137.74.169.241

As we can see, the IP address changes every time in the sessionless case. However, in the case where we use a session object, the IP address only changes for the very first request and all subsequent requests show the same IP address indicating that the circuit did not change.

Why does this happen and how can it be resolved? How can I use request sessions without losing the ability to switch circuits?

maybe create new session for new IP. One session for different IPs may make no sense. User with new IP should get new cookies, etc. from server so it should start new, clear session. — furas, Jan 05 '18 at 17:59

score 2 · Accepted Answer · answered Jan 05 '18 at 22:11

2

This is because of...

Keep-Alive

Excellent news — thanks to urllib3, keep-alive is 100% automatic within a session! Any requests that you make within a session will automatically reuse the appropriate connection!

Note that connections are only released back to the pool for reuse once all body data has been read; be sure to either set stream to False or read the content property of the Response object.

When you use a session, it sends a Connection: keep-alive header and will hang onto the connection which causes it to use the previously established circuit in Tor.

As mentioned in the docs, you can set stream=False to get around it:

http.get("https://httpbin.org/ip", timeout=(5, 5), proxies=proxies, stream=False)

answered Jan 05 '18 at 22:11

drew010

68,777
11
134
162

You are right in that Keep-Alive is the cause of this behavior. However, your solution, i.e., setting `stream` to `False`, did not solve my problen. It does not disable Keep-Alive (and it shouldn't do that if I read the docs correctly). To get around it I had to explicitly set the `Connection` header to `close`. Maybe you want to add that to your answer. – user1658887 Jan 05 '18 at 23:04
@user1658887, would you mind clarifing what you mean for "To get around it I had to explicitly set the Connection header to close. Maybe you want to add that to your answer."? I'm just a student on this and I would love to learn how to solve this issue – Glori P. Jan 26 '21 at 17:24

Python stem+requests: Not switching circut/changing IP address when using a session

1 Answers1