How do you test a function using which retrieves data by urllib2?

Question

I am getting into testing in python and I asked myself how to test this method.

def get_response(self, url, params):
    encoded_params = urllib.urlencode(params)
    request = urllib2.Request(BASE_URL, headers=HEADERS)
    response = urllib2.urlopen(request, encoded_params)
    return response

Using doctest or unittest, how is this best achieved? I thought of passing get_response() a test url and some test parameters, which exists in real world and to check if response.read() returns the expected data. But somehow I feel, this is not the way it should be done. Any suggestions? I would appreciate suggestions how to handle such cases in context of texting.

First, define the contract the method should adhere to. The post outlines testing the *accepted response*, but now how failures should be handled. Testing the actual *data*, however, sounds like more of an integration test. — , Jul 15 '11 at 21:06
Have a look at the (BSD-licensed) scikit-learn code that does exactly this; [testcases](https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/datasets/tests/test_mldata.py), [`mock_urllib2`](https://github.com/scikit-learn/scikit-learn/blob/master/scikits/learn/utils/testing.py#L55). — Fred Foo, Jul 15 '11 at 21:14

score 5 · Answer 1 · answered Jul 15 '11 at 21:09

This is a good case for using fake objects:

# my_module
get_url = urllib2.urlopen

def get_response(self, url, params):
    encoded_params = urllib.urlencode(params)
    request = urllib2.Request(BASE_URL, headers=HEADERS)
    response = get_url(request, encoded_params)
    return response

# test_my_module    
def fake_get_url(request, params):
    assert request == "the url I expect"
    assert params == ['the', 'params', 'i', 'expect']
    return SomeFakeResponse("OK")

my_module.get_url = fake_get_url
assert my_module.get_response("blah", "blah").content == "OK"

This is just the sketchiest sketch of how you could override the real urllib2.urlopen function with your own fake implementation to test your code without really hitting the web.

SingleNegationElimination · Accepted Answer · 2011-07-16T03:37:24.203

This is a good opportunity to use a mock testing framework, such as minimock.

BASE_URL='http://google.com'
HEADERS = {"accept-language":"klingon"}
import urllib, urllib2
def get_response(self, url, params):
    r"""
    >>> from minimock import Mock
    >>> urllib2.Request = Mock('urllib2.Request')
    >>> urllib2.urlopen = Mock('urllib2.urlopen')
    >>> get_response(None, 'http://google.com', {'foo':'bar'})
    Called urllib2.Request(
        'http://google.com',
        headers={'accept-language': 'klingon'})
    Called urllib2.urlopen(None, 'foo=bar')
    >>> 
    """
    encoded_params = urllib.urlencode(params)
    request = urllib2.Request(BASE_URL, headers=HEADERS)
    response = urllib2.urlopen(request, encoded_params)
    return response

Do note that the unit is embedded in the docstring for the function under test, in doctest format.

How do you test a function using which retrieves data by urllib2?

2 Answers2