How can I use Test::LWP::UserAgent if I cannot replace the $ua in the app code directly?

Question

I've got a sub that retrieves some data from an API via a REST service. The code is rather simple, but I need to post parameters to the API and I need to use SSL, so I have to go through LWP::UserAgent and cannot use LWP::Simple. This is a simplified version of it.

sub _request {
  my ( $action, $params ) = @_;

  # User Agent fuer Requests
  my $ua = LWP::UserAgent->new;
  $ua->ssl_opts( SSL_version => 'SSLv3' );

  my $res = $ua->post( 
    $url{$params->{'_live'} ? 'live' : 'test'}, { action => $action, %$params } 
  );
  if ( $res->is_success ) {
    my $json = JSON->new;

    return $json->decode( $res->decoded_content );
  } else {
    cluck $res->status_line;
    return;
  }
}

This is the only place in my module (which is not OOp) where I need the $ua.

Now I want to write a test for this and after some research decided it would be best to use Test::LWP::UserAgent, which sounds really promissing. Unfortunately, there's a catch. In the doc, it says:

Note that LWP::UserAgent itself is not monkey-patched - you must use this module (or a subclass) to send your request, or it cannot be caught and processed.

One common mechanism to swap out the useragent implementation is via a lazily-built Moose attribute; if no override is provided at construction time, default to LWP::UserAgent->new(%options).

Arghs. Obviously I cannot do the Moose thing. I can't just pass a $ua to the sub, either. I could of course add an optional third param $ua to the sub, but I don't like the idea of doing that. I feel it's not ok to alter the behaviour of such simple code so radically just to make it testable.

What I basically want to do is run my test like this:

use strict;
use warnings;
use Test::LWP::UserAgent;
use Test::More;

require Foo;

Test::LWP::UserAgent->map_response( 'www.example.com',
  HTTP::Response->new( 200, 'OK', 
    [ 'Content-Type' => 'text/plain' ], 
    '[ "Hello World" ]' ) );

is_deeply(
  Foo::_request('https://www.example.com', { foo => 'bar' }),
  [ 'Hello World' ],
  'Test foo'
);

Is there a way to monkeypatch the Test::LWP::UserAgent functionality into LWP::UserAgent so that my code just uses the Test:: one?

IMO, testability is a valid code concern. The goal of coding is to have something that "works". So, a statement of the target state would go like this: "it works". However, experience has taught me this: "Nothing 'works' until testing says it does." — Axeman, Feb 19 '13 at 16:49

score 4 · Answer 1 · answered Feb 19 '13 at 19:21

4

Change your code so that within _request(), you're calling _ua() to collect your user agent and override this method in your test script. Like so:

Within your module:

sub _request {
...
 my $ua = _ua();
...
}

sub _ua { 
 return LWP::UserAgent->new();
}

Within your test script:

...
Test::More::use_ok('Foo');

no warnings 'redefine';
*Foo::_ua = sub { 
    # return your fake user agent here
};
use warnings 'redefine';
... etc etc

answered Feb 19 '13 at 19:21

Petski

341
3
4

Thanks for the suggestion. I like @rjh's approach better, though. I don't want to change too much. – simbabque Feb 20 '13 at 10:04

score 2 · Accepted Answer · answered Feb 19 '13 at 16:27

I could of course add an optional third param $ua to the sub, but I don't like the idea of doing that. I feel it's not ok to alter the behaviour of such simple code so radically just to make it testable.

This is known as dependency injection and it's perfectly valid. For testing you need to be able to override objects your class will use to mock various results.

If you prefer a more implicit way of overriding objects, consider Test::MockObject and Test::MockModule. You could mock LWP::UserAgent's constructor to return a test object instead, or mock a wider part of the code you are testing such that Test::LWP::UserAgent is not needed at all.

Another approach is to refactor your production code such that the components are (unit) testable in isolation. Split the HTTP fetching from the processing of the response. Then it is very simple to test the second part, by creating your own response object and passing it in.

Ultimately programmers use all of the above tools. Some are appropriate for unit testing and others for wider integration testing.

Thanks for the suggestion. When I read what the T::LWP::UA doc says in the [motivation section](https://metacpan.org/module/Test::LWP::UserAgent#MOTIVATION) (*Most mock libraries on the CPAN use Test::MockObject, which is widely considered not good practice...*) I sort of forgot about these two for this test. I'm using MockModule a lot, but somehow that slipped my mind completely here. What Test::LWP::UserAgent does just sounds to great to think about fiddling yourself. ;-) — simbabque, Feb 20 '13 at 08:55

score 0 · Answer 3 · edited May 23 '17 at 10:29

As of today, I would go with the following approach to this problem. Imagine this piece of legacy code¹, that is not object oriented and cannot be refactored so that it makes dependency injection easy.

package Foo;
use LWP::UserAgent;

sub frobnicate {
    return LWP::UserAgent->new->get('http://example.org')->decoded_content;
}

This is indeed tricky to test, and rjh's answer is spot-on. But in 2016 we have a few more modules available than we did back in 2013. I particularly like Sub::Override, which replaces a sub in a given namespace, but only keeps it around in the current scope. That makes it great for unit tests, because you don't need to care about restoring everything after you're done.

package Test::Foo;
use strict;
use warnings 'all';
use HTTP::Response;
use Sub::Override;
use Test::LWP::UserAgent;
use Test::More;

# create a rigged UA
my $rigged_ua = Test::LWP::UserAgent->new;
$rigged_ua->map_response( 
    qr/\Qexample\E/ => HTTP::Response->new( 
        '200', 
        'OK', 
        [ 'Content-Type' => 'text/plain' ], 
        'foo',
    ), 
);

# small scope for our override
{
    # make LWP return it inside our code
    my $sub = Sub::Override->new( 
        'LWP::UserAgent::new'=> sub { return $rigged_ua } 
    );
    is Foo::frobnicate(), 'foo', 'returns foo';
}

We basically create a Test::LWP::UserAgent object that we feed all of our test cases. We can also give it a code ref that will run tests on the request if we want (not shown here). We then use Sub::Override to make LWP::UserAgent's constructor not return an actual LWP::UA, but the already prepared $rigged_ua. Then we run our test(s). Once $sub goes out of scope, LWP::UserAgent::new gets restored and we do not interfere with anything else.

It's important to always do those tests with the smallest possible scope (as most things in Perl).

If there are a lot of those test cases, it's a good strategy to build some kind of configuration hash for what you expect for each request, and use a building helper function to create the rigged user agent, and another one to create the Sub::Override object. Used in a lexical scope, this approach is very powerful and pretty concise at the same time.

^{1) represented here by the lack of use strict and use warnings.}

How can I use Test::LWP::UserAgent if I cannot replace the $ua in the app code directly?

3 Answers3