35

I have been able to view the attributes of the PreparedRequest that botocore sends, but I'm wondering how I can view the exact request string that is sent to AWS. I need the exact request string to be able to compare it to another application I'm testing AWS calls with.

zachhilbert
  • 639
  • 1
  • 5
  • 12

2 Answers2

73

You could also enable debug logging in boto3. That will log all requests and responses as well as lots of other things. Its a bit obscure to enable it:

import boto3
boto3.set_stream_logger(name='botocore')

The reason you have to specify botocore as the name to log is that all of the actual requests and responses happen at the botocore layer.

garnaat
  • 44,310
  • 7
  • 123
  • 103
  • I tried that before using the proxy. The problem is that it doesn't log the actual full request as sent to the server, only the components it uses to calculate signature, service, etc. – zachhilbert May 16 '15 at 01:03
  • 2
    I've had success with this technique when debugging dynamodb requests; I was able to find the entire request body in the log along with all the responses. – killthrush May 09 '19 at 13:21
  • 7
    You can also use `import logging` `logging.getLogger('botocore').setLevel(logging.DEBUG)` to the same effect. – Craig Ringer Jun 17 '19 at 07:37
8

So what you probably want to do is to send your request through the proxy (mitmproxy, squid). Then check the proxy for what was sent. Since HTTPS data is encrypted you must first decrypt it, then log the response, then encrypt it back and send to AWS. One of the options is to use mitmproxy. ( It's really easy to install )

  1. Run mitmproxy

  2. Open up another terminal and point proxy to mitmproxys port:

    export http_proxy=127.0.0.1:8080
export https_proxy=$http_proxy

  3. Then set verify=False when creating session/client

    In [1]: import botocore.session

In [2]: client = botocore.session.Session().create_client('elasticache', verify=False)

  4. Send request and look at the output of mitmproxy

    In [3]: client.describe_cache_engine_versions()

  5. The result should be similar to this:

    Host:             elasticache.us-east-1.amazonaws.com
Accept-Encoding:  identity
Content-Length:   53
Content-Type:     application/x-www-form-urlencoded
Authorization:    AWS4-HMAC-SHA256 Credential=FOOOOOO/20150428/us-east-1/elasticache/aws4_request, SignedHeaders=host;user-agent;x-amz-date, Signature=BAAAAAAR
X-Amz-Date:       20150428T213004Z
User-Agent:       Botocore/0.103.0 Python/2.7.6 Linux/3.13.0-49-generic

<?xml version='1.0' encoding='UTF-8'?>
<DescribeCacheEngineVersionsResponse
xmlns="http://elasticache.amazonaws.com/doc/2015-02-02/">  
<DescribeCacheEngineVersionsResult>
    <CacheEngineVersions>
      <CacheEngineVersion>
      <CacheParameterGroupFamily>memcached1.4</CacheParameterGroupFamily>
    <Engine>memcached</Engine>
    <CacheEngineVersionDescription>memcached version 1.4.14</CacheEngineVersionDescription>
    <CacheEngineDescription>memcached</CacheEngineDescription>
    <EngineVersion>1.4.14</EngineVersion>
Vor
  • 33,215
  • 43
  • 135
  • 193
  • I wasn't able to get mitmproxy quickly working using pip, BUT I used your idea and ran the request with Burp Suite. I was able to see where my error was coming from. Thank you for the idea! – zachhilbert Apr 28 '15 at 22:49