0

I'm writing some kind of a recursive parser. The simplest form is:

  1. Take all links from first link's page body
  2. Repeat the first step for each link

So now I want to test it. The problem is I can't figure out the best way to mock all these pages. I use http package and I already have some tests written using httptest package (via httptest.NewServer). But it seems to be no use for my task now. I guess the best way is to use http.Client with custom Transport struct, but it's lots of boilerplate and additional smelly code. Is there a more elegant way to do this?

Jonathan Hall
  • 75,165
  • 16
  • 143
  • 189
Stanislav
  • 47
  • 2
  • 5
  • What's the problem with using `httptest`'s server for this? – Adrian Feb 20 '20 at 19:00
  • @Adrian , you mean to create 5-10 fake servers for each url? Another problem with this solution is it should return some html with these links, and I can only get each link after each server is created =\ – Stanislav Feb 21 '20 at 07:41
  • Why would you need a server per URL? – Adrian Feb 21 '20 at 14:13
  • Mmm... Because I need different urls. What are your suggestions then if not `Transport` mocking? – Stanislav Feb 21 '20 at 15:17
  • Any HTTP server can handle many URLs, otherwise they'd be pretty useless. I still don't see why you'd need multiple test servers. – Adrian Feb 21 '20 at 15:20
  • Because our `http.Client` will call real websites and not our mocked ones? We can't force it to only send requests via our mock server. I'm not experienced with http package in golang (and golang in general), your questions are a bit frustrating. – Stanislav Feb 22 '20 at 13:04

1 Answers1

1

I have used a custom Transport a couple of times for testing clients. Usually I would create some helper types and functions to cut down on boilerplate code.

Something like this could be a start.

package main

import (
    "bytes"
    "fmt"
    "io"
    "io/ioutil"
    "net/http"
    "os"
)

type roundTripFunc func(*http.Request) (*http.Response, error)

func (r roundTripFunc) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    return r(req)
}

func main() {
    c := &http.Client{
        Transport: roundTripFunc(func(req *http.Request) (resp *http.Response, err error) {
            return &http.Response{
                StatusCode: 200,
                Body:       ioutil.NopCloser(bytes.NewBufferString("test")),
            }, nil
        }),
    }

    r, _ := c.Get("/")
    fmt.Printf("%#v\n", r)
    io.Copy(os.Stdout, r.Body)
}

For example if your testing a JSON API client you could make a round trip helper function takes care of decoding, encoding, headers etc. In your case maybe you could make the round trip function map host header plus URL path into file fixtures paths?

Mattias Wadman
  • 11,172
  • 2
  • 42
  • 57