urequests only working on specific websites

Question

I am using a nodemcu esp8266 and the plan is to make a spotify api request to automate playing some songs, and I have a fully functional python script that does exactly that, but when I tried to convert it to upython it failed. I have now spent hours on this and have figured out that the problem is that the urequest for some reason only works on specific websites, for example if I try:

import urequests as requests

x = requests.get('https://www.spotify.com')
print(x.status_code)

i get a 302

but when I try:

import urequests as requests

x = requests.get('https://www.google.com')
print(x.status_code)

I get a 200. That problem percists with a few websites and with no valid response my urequests.put command does not return the things I need... any ideas how to fix it? thank you in advance.

this is the code I am trying to run:

import urequests as requests
import ujson as json 

refresh_token = "xxxxx"
base_64 = "xxxxx"


class Refresh:

def __init__(self):
    self.refresh_token = refresh_token
    self.base_64 = base_64

def refresh(self):

    query = "https://accounts.spotify.com/api/token"
    
    payload={"grant_type": "refresh_token", "refresh_token": refresh_token}
    headers={'Authorization': 'Basic %s' % base_64}
    data = (json.dumps(payload)).encode()
    
    response = requests.post(query,
                              data=data,
                              headers=headers)


    print(response.status_code)
    print(response.reason)

a = Refresh()
a.refresh()

That's not a problem with `urequests`. That's how the websites work. E.g., try `curl -i https://www.spotify.com`; that sends a redirect (if you're in the US) to `https://www.spotify.com/us/`. What you're seeing is the expected behavior of the website you're trying to access. — larsks, May 25 '22 at 20:11
but I have the exact same code in normal python with the normal request function and that works perfectly fine. plus that problem is for a microcontroller and as far as I can tell it only has micropython installed and not curl — Jonny Morgan, May 25 '22 at 21:03
The normal `requests` module follows redirects by default. `urequests` does not; you would need to parse the 302 response and extract the `Location` header yourself. I was not suggesting running `curl` on the microcontroller; I was suggesting that running on your host to demonstrate to yourself that `urequests` is accurately presenting the response from the website. — larsks, May 25 '22 at 22:01
ohhhh thats what you mean.. well when I run curl it redirects me to https://www.spotify.com/at/, the only problem is I have already tried putting that address into the urequests and it gives me a status code 500. If I am missing something I am really sorry, I have been searching for a solution for so many hours now and I am exhausted... plus I am still kinda new to working with requests so im sorry I you have to spell out something obvious — Jonny Morgan, May 25 '22 at 22:34
The content at `https://www.spotify.com/at/` is over 62k; I suspect you're running into a memory error on the esp8266. But if you'll be using the [spotify API](https://developer.spotify.com/documentation/web-api/reference/#/), you wouldn't be connecting to `https://www.spotify.com/at/` and reading that content; you would be connecting to the API endpoint `https://api.spotify.com/v1`. — larsks, May 25 '22 at 22:51
yes wow that was exactly what was happening, I tried the new adress and ran into a 400 bad request error message and the same with the api.spotify so apparently it was a problem with my request after all.. I tried so many different versions and thought I had fixed that problem but apparently i never found the correct solution.. I edited the question to provide the token request code I am trying to run, could you please tell me what I have to fix to get it to work? Thank you so much for your answers so far, they have helped me so much!!! — Jonny Morgan, May 26 '22 at 13:20

score 1 · Accepted Answer · answered May 27 '22 at 01:59

There are several things going on here that are going to cause your problems.

First, you're trying to send JSON data to the /api/token endpoint:

    data = (json.dumps(payload)).encode()
    
    response = requests.post(query,
                              data=data,
                              headers=headers)

...but according to the documentation, this endpoint excepts application/x-www-form-urlencoded data.

Second, while the documentation suggests you need to send a basicAuthorization header, in my experiments this evening I wasn't able to get that to work...but I was able to successfully refresh a token if I included the client id and secret in the request body.

You can see my forum post with this question here.

With that in mind, the following code seems to work on my esp8266 running micropython 1.18:

import urequests as requests

refresh_token = "..."
client_id = "..."
client_secret = "..."


class Refresh:
    def __init__(self, refresh_token, client_id, client_secret):
        self.refresh_token = refresh_token
        self.client_id = client_id
        self.client_secret = client_secret

    def refresh(self):
        url = "https://accounts.spotify.com/api/token"
        data = "&".join(
            [
                "grant_type=refresh_token",
                f"refresh_token={self.refresh_token}",
                f"client_id={self.client_id}",
                f"client_secret={self.client_secret}",
            ]
        )

        headers = {
            "content-type": "application/x-www-form-urlencoded",
        }

        response = requests.post(url, data=data, headers=headers)

        print(data)
        print(response.status_code)
        print(response.reason)
        print(response.text)


a = Refresh(refresh_token, client_id, client_secret)
a.refresh()

urequests only working on specific websites

1 Answers1