4

I am writing an HTTP record-playback proxy which internally uses WireMock, and am having a problem with recording HTTPS targets. HTTP sites work fine.

Here is how I set up the WireMock proxy for an unencrypted site:

java \
    -jar /var/proximate/wiremock-standalone-2.4.1.jar \
    --port 9000 \
    --proxy-all http://ilovephp.jondh.me.uk \
    --record-mappings \
    --root-dir /remote/experiment/record/http

I then can record anything on that site using this command:

wget -e use_proxy=yes -e http_proxy=proximate-proxy:9000 \
    http://ilovephp.jondh.me.uk/en/tutorial/make-your-own-blog

In short order I will get some automatically created mappings:

/ # ls -R /remote/experiment/record/http
/remote/experiment/record/http:
__files   mappings

/remote/experiment/record/http/__files:
body-tutorial-make-your-own-blog-tWrNm.txt

/remote/experiment/record/http/mappings:
mapping-tutorial-make-your-own-blog-tWrNm.json

Here is much the same thing, but on an SSL site. Proxy first:

java \
    -jar /var/proximate/wiremock-standalone-2.4.1.jar \
    --port 9000 \
    --proxy-all https://www.rottentomatoes.com/ \
    --record-mappings \
    --root-dir /remote/experiment/record/https \
    --verbose

And here is the fetch:

wget -e use_proxy=yes -e http_proxy=proximate-proxy:9000 \
    https://www.rottentomatoes.com/

The result from WireMock is that the internal cache directories are created, but they do not contain anything:

/ # ls -R /remote/experiment/record/https
/remote/experiment/record/https:
__files   mappings

/remote/experiment/record/https/__files:

/remote/experiment/record/https/mappings:

I am using WM 2.4.1 on Alpine 3.4 in a Docker container. Do I need to use some of the https-specific command line switches to get this to work?

Update

I was temporarily heartened to see that recent release 2.5.0 had a recent merge of https-bind-address-bug, which sounded very relevant. I have tried with this release, but unfortunately it makes no difference.

halfer
  • 19,824
  • 17
  • 99
  • 186

1 Answers1

3

This is partly solved - my main error was that the proxy address for HTTPS connections is set separately, for both WireMock and the client. I am now using:

java \
    -jar /var/proximate/wiremock-standalone-2.4.1.jar \
    --port 9000 \
    --https-port 9050 \
    --proxy-all https://www.rottentomatoes.com/ \
    --record-mappings \
    --root-dir /remote/experiment/record/https \
    --verbose \
    --print-all-network-traffic

I've added --print-all-network-traffic so I can see if any HTTPS traffic is hitting the proxy at all (it is now).

Here is the new client command, complete with new proxy directive:

wget -e use_proxy=yes -e http_proxy=proximate-proxy:9000 \
     -e https_proxy=proximate-proxy:9050 \
     --verbose \
     --user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0" \
     https://www.rottentomatoes.com/

However, now the proxy is actually being used, it looks like the operation is not completing successfully. I've tried other encrypted sites and received the same vague errors:

Resolving proximate-proxy... 172.18.0.2
Connecting to proximate-proxy|172.18.0.2|:9050... connected.
Failed reading proxy response: No error information
Retrying.

--2017-01-26 09:20:18--  (try: 2)  https://www.rottentomatoes.com/
Connecting to proximate-proxy|172.18.0.2|:9050... connected.
Failed reading proxy response: No error information
Retrying.

--2017-01-26 09:20:20--  (try: 3)  https://www.rottentomatoes.com/
Connecting to proximate-proxy|172.18.0.2|:9050... connected.
Failed reading proxy response: No error information
Retrying.

^C

I have replaced the user agent in case there is some third-party rejection based on the standard "wget" string, but this does not seem to affect things either.

I will ask a new question about this, to narrow down whether this is a WireMock or a Wget problem.

halfer
  • 19,824
  • 17
  • 99
  • 186