headers and oauth in get_quote for E*Trade API using Python3

Question

After authorizing my application, I request an access token by passing the oauth credentials through the header. The signature and headers are generated through this code;

API_module = 'oauth'
API_RESTful = 'access_token'
if renewal:
    API_RESTful = 'renew_access_token'
production_url = 'https://etws.etrade.com/{0:s}/{1:s}'.format(API_module, API_RESTful)

oauth_timestamp = int(time.time())
rand_str = lambda n: ''.join([random.choice(string.hexdigits) for i in range(n)])
oauth_nonce = rand_str(40)
key = oauth_consumer_secret + \
      '&' + \
      quote_plus(oauth_token_secret)
base_string = quote_plus('GET') + '&' + \
              quote_plus('https://etws.etrade.com/oauth/access_token') + '&' + \
              quote_plus('oauth_consumer_key={}&'.format(oauth_consumer_key)) + \
              quote_plus('oauth_nonce={}&'.format(oauth_nonce)) + \
              quote_plus('oauth_signature_method=HMAC-SHA1&') + \
              quote_plus('oauth_timestamp={:d}&'.format(oauth_timestamp)) + \
              quote_plus('oauth_token={}&'.format(quote_plus(oauth_token))) + \
              quote_plus('oauth_verifier={}'.format(oauth_verification_code))
hashed = hmac.new(key.encode(), base_string.encode(), sha1)
oauth_signature = quote_plus(binascii.b2a_base64(hashed.digest())[:-1])
header_string = 'Authorization: OAuth ' + \
                'realm="",' + \
                'oauth_signature="{}",'.format(oauth_signature) + \
                'oauth_nonce="{}",'.format(quote_plus(oauth_nonce)) + \
                'oauth_signature_method="{}",'.format(oauth_signature_method) + \
                'oauth_consumer_key="{}",'.format(oauth_consumer_key) + \
                'oauth_timestamp="{}",'.format(str(oauth_timestamp)) + \
                'oauth_verifier="{}",'.format(oauth_verification_code) + \
                'oauth_token="{}"'.format(quote_plus(oauth_token))
headers_list.append(header_string)

response = curl_get_http(current_url=production_url)

produces these headers;

Host: etws.etrade.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: keep-alive
Authorization: OAuth
realm="",
oauth_signature="fzqLbI8LBlBGs1Clp4eAgs09YuM%3D",
oauth_nonce="E447Ea1FCfbcCF0116fbdC47bE8E4aA4Cf7e3Aab",
oauth_signature_method="HMAC-SHA1",
oauth_consumer_key="4b6471c7ee",
oauth_timestamp="1501003943",
oauth_verifier="O5K2A",
oauth_token="BVuKV9Q7F93OxjbqY%2FzRmoqI0M%3D"

the request returns;

oauth_token=3W3hs5aSQPwMR%2FM0H0%2BPWhI%2Bo%3D&oauth_token_secret=SWvknmgEIgKzbN35bwwoNw%3D

The key is updated by replacing the old token_secret with the new token_secret. The base string is also updated with the new token value and with a new time stamp and nonce. These new values are used to generate a new signature.

url_quotes = 'https://etws.etrade.com/market/rest/quote/{0:s}?detailFlag={1:s}'.format(symbols, flag)

oauth_timestamp = int(time.time())
rand_str = lambda n: ''.join([random.choice(string.hexdigits) for i in range(n)])
oauth_nonce = rand_str(40)

key = oauth_consumer_secret + \
      '&' + \
      oauth_token_secret
base_string = quote_plus('GET') + '&' + \
              quote_plus('https://etws.etrade.com/market/rest/quote') + '&' + \
              quote_plus('oauth_consumer_key={}&'.format(oauth_consumer_key)) + \
              quote_plus('oauth_nonce={}&'.format(oauth_nonce)) + \
              quote_plus('oauth_signature_method=HMAC-SHA1&') + \
              quote_plus('oauth_timestamp={:d}&'.format(oauth_timestamp)) + \
              quote_plus('oauth_token={}&'.format(oauth_token)) #+ \
              #quote_plus('oauth_verifier={}'.format(oauth_verification_code))
hashed = hmac.new(key.encode(), base_string.encode(), sha1)
oauth_signature = quote_plus(binascii.b2a_base64(hashed.digest())[:-1])
header_string = 'Authorization: OAuth ' + \
                'realm="",' + \
                'oauth_signature="{}",'.format(oauth_signature) + \
                'oauth_nonce="{}",'.format(quote_plus(oauth_nonce)) + \
                'oauth_signature_method="{}",'.format(oauth_signature_method) + \
                'oauth_consumer_key="{}",'.format(oauth_consumer_key) + \
                'oauth_timestamp="{:d}",'.format(oauth_timestamp) + \
                'oauth_verifier="{}",'.format(oauth_verification_code) + \
                'oauth_token="{}"'.format(oauth_token)
headers_list.append(header_string)
response = curl_get_http(current_url=url_quotes)

Changing the headers to;

Host: etws.etrade.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: keep-alive
Authorization: OAuth
realm="",
oauth_signature="v4xa%2FrCKtFRSUdHw%3D",
oauth_nonce="57FCC260F81b2fAd95AccA69FE07BFFcd06d83AB",
oauth_signature_method="HMAC-SHA1",
oauth_consumer_key="4b6471c7ee",
oauth_timestamp="1501003945",
oauth_verifier="O5K2A",
oauth_token="3W3hs5aSQPwMR%2FM0H0%2BPWhI%2Bo%3D"

and a get_quote request is made;

https://etws.etrade.com/market/rest/quote/TICK,ER,THAT,GOES,UP?detailFlag=FUNDAMENTAL

Though, instead of a quote, an oauth problem is returned.

<Error>
<message>oauth_problem=signature_invalid</message>
</Error>

I have tried passing the information in the url, but it returns the same error. Are there procedural errors in the request? Should the new token be used without updating the signature?

(the posted credentials have been changed to protect the innocent)

[Edit] your Question and show your code from `OAuthSession` and `request.post(...). — stovfl, Jul 25 '17 at 19:33
Thanks. Just fyi, I switched to TD Ameritrade for this. The API is much friendlier, and also has a pretty good system for historical data. — jsfa11, Nov 25 '20 at 12:34

kingdc · Accepted Answer · 2017-07-26T00:33:15.877

The problem is in the signature generation.

For the URL portion, you should include the full URL up to the query string. In this example it would be:

base_string = quote_plus('GET') + '&' + \
          quote_plus('https://etws.etrade.com/market/rest/quote/TICK,ER,THAT,GOES,UP') + '&' + \

When concatenating the oauth parameters you should include all non-Oauth query parameters as well. They need to be in sorted order per the spec (https://oauth.net/core/1.0a/#anchor13). In this case you would need to include detailFlag=FUNDAMENTAL as your first parameter:

quote_plus('detailFlag=FUNDAMENTAL&') + \ 
quote_plus('oauth_consumer_key={}&'.format(oauth_consumer_key)) + \

Also, the oauth_verifier is not needed in the header except to retrieve the access token.

I would not have gotten the parameter's inclusion to the base string. Thanks! At first it did not work and I read through the OAuth Core 1.0a and the [RFC3986] again to make sure I had not overlooked something again. Finally, I found the trailing ampersand in the base string. I deleted the single character and saw quotes! I'll play around with XML versus JSON tomorrow, but Thanks for the help. — jsfa11, Jul 26 '17 at 05:24

headers and oauth in get_quote for E*Trade API using Python3

1 Answers1