How to get around a 401 unauthorized error?

Question

I'd like to output some values from this site. With the browser inspector I located the URL to the JSON that has that information that I want. Once with the JSON, I can extract the values just fine.

The issue is that the URL to the JSON only works for a limited amount of time. If I try to access it later (via browser or via script's UrlFetch), I get a:

{
"status": 401,
"response": "unauthorized"
}

Since I would like to check on the target JSON periodically, I would like to find a way to authorize the request. Any ideas?

Thanks!

PS: This is what I've tried so far (google apps script):

function getUserAndJSON(){
  var url = 'https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc';

  var opt = {
    "method": "GET", 
    "muteHttpExceptions": true,
    "escaping":false,
    "headers":JSON.parse(user)
  }
  var str = UrlFetchApp.fetch(url,opt).getContentText();
  Logger.log(str); //error 401
}

can u tell where is this url located on the site, I need to see it working and then can inspect the request header it's sending. As u can see that ur url is having endpoint and body but to see headers you need to see the request header. U can do it in browser also by opeing ur network window and the see where this api is making the request. Let me know if this works or any other help is needed. — 1UC1F3R616, Apr 05 '20 at 10:17
@1UC1F3R616 The url is the first link on the question under 'this site'. I have opened up the network tab under the dev tools, but I fail to find the file where the request header is being sent. Help would be appreciated. Cheers — jlo, Apr 06 '20 at 19:13
problem is I don't see where is this question, the whole site is in some other language. Open under network tab, then click on any api, now on side there will appear options like Response, Request Header and so on. You click them and then inspect it. — 1UC1F3R616, Apr 09 '20 at 08:38
Please provide the entire 401 status response including headers. This will give you a hint regarding the type of authentication and then you can work with it to generate a token every time you get a 401 message. An example of a fairly use authentication mechanism: https://en.wikipedia.org/wiki/Basic_access_authentication — ofirule, Apr 09 '20 at 21:06
@1UC1F3R616 here is a screenshot of my network screen, could you point out where I can find the options 'any API' you mention? https://postimg.cc/Lq1hwvss — jlo, Apr 11 '20 at 12:26
@jlo click on them and then a window will pop up on side, I am uisng Firefox. Use Postman to test apis, in python there is a request module u can send headers and cookies in it. — 1UC1F3R616, Apr 11 '20 at 13:49

score 5 · Accepted Answer · answered Apr 12 '20 at 10:20

One solution to this problem is to simply use the option 'copy as... > fetch' within the browser inspector (in my case Opera), which gives out the following (somewhat different headers than the ones sent by the browser):

fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc", {"credentials":"omit","headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});

Which I used in GAS in the following way:

function testGetJSON(){
 var str = UrlFetchApp.fetch("https://asunnot.oikotie.fi/api/cards?cardType=100&conditionType%5B%5D=1&conditionType%5B%5D=2&limit=24&locations=%5B%5B1669,4,%22Lauttasaari,+Helsinki%22%5D,%5B14714,5,%2200340,+Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&offset=0&price%5Bmax%5D=600000&price%5Bmin%5D=150000&roomCount%5B%5D=3&size%5Bmin%5D=35&sortBy=published_sort_desc",
 {"credentials":"omit",
"headers":{"accept":"application/json","ota-cuid":"fd2a3a03d52a2721f9a9aa844ddf7eef2ac66ed6","ota-loaded":"1586685082","ota-token":"ab7e9f830a7dff3a9b01fbdcbc899ed7bfa659a4793103f1943e83ef5f938b16","sec-fetch-dest":"empty"},"referrer":"https://asunnot.oikotie.fi/myytavat-asunnot?conditionType%5B%5D=1&conditionType%5B%5D=2&locations=%5B%5B1669,4,%22Lauttasaari,%20Helsinki%22%5D,%5B14714,5,%2200340,%20Helsinki%22%5D%5D&lotOwnershipType%5B%5D=1&price%5Bmax%5D=600000&price%5Bmin%5D=150000&size%5Bmin%5D=35&roomCount%5B%5D=3&cardType=100","referrerPolicy":"no-referrer-when-downgrade","body":null,"method":"GET","mode":"cors"});
  Logger.log(str.getContentText());
}

Which works perfectly.

Thank you all for your tips and suggestions!

How to get around a 401 unauthorized error?

1 Answers1