0

Need help with a 407 response? I can't solve this problem. I had a parser for a Korean car site that worked daily for one month until it showed a 407 error. I googled that the problem is solved by proxy substitution, but either I'm doing something wrong, or the proxy does not fix the problem.

Sample code to test:

import requests

proxies = {
    'https': 'http://user:pass@xx.xxx.xx.xxx:xxxx'
}
url = 'https://api.encar.com/search/car/list/mobile?count=true&q=(And.Hidden.N._.(C.CarType.N._.Manufacturer.%EB%A7%88%EC%AF%94%EB%8B%A4.))&sr=%7CModifiedDate%7C0%7C200&inav=%7CMetadata%7CSort'
response = requests.get(url, proxies=proxies)

print(response.status_code)
mikholand
  • 1
  • Code 407 means, a lack of authentication credentials, so most likely the username and password that you're adding to the proxy aren't working. – Zero Jul 19 '23 at 11:56
  • @Zero Without a proxy, I get a 407 response, and when I add a working proxy (I checked on other sites), I get the same response. – mikholand Jul 19 '23 at 12:04
  • Then the problem is your main request then, the API might be expecting some credentials to show you the information. – Zero Jul 19 '23 at 12:06
  • @Zero When I pull this link from the site, it opens in the browser and gives the normal result that I need. – mikholand Jul 19 '23 at 12:09
  • Try incognito, just browsers have so many elements which you do have control over, like cookies, cache etc. – Zero Jul 19 '23 at 12:14
  • @Zero So in the browser, on the contrary, everything is fine. When I run the python code, then requests gives a 407 response – mikholand Jul 19 '23 at 12:38
  • That's what I mean, try using the browser in Incognito mode, or try Postman. – Zero Jul 19 '23 at 12:40
  • @Zero Incognito mode and Postman gives a 407 response – mikholand Jul 19 '23 at 13:28
  • Exactly, that's because in the main browser where you're able to see the results, has cookies attached. – Zero Jul 19 '23 at 13:54
  • @Zero Found this feature. When I substitute the User-Agent on my local computer it returns 200, but as soon as I upload the code to the server, it returns 407 again. – mikholand Jul 19 '23 at 13:58
  • Try to replicate the requests in the Postman and see what works for you – Zero Jul 19 '23 at 14:04
  • @Zero That's exactly what I did. Postman gives 407 with a clean request. When I add a host and a user agent to the header, the request is processed normally and gives 200. But with the same parameters on the server it returns 407 – mikholand Jul 19 '23 at 14:19
  • I think I wasn't able to explain properly, that is the exact purpose of Postman, if a set of headers work for you, then use the same in your code. – Zero Jul 19 '23 at 14:43
  • @Zero I upload to the server the same headers that I use in postman – mikholand Jul 19 '23 at 15:16
  • any issues so far? @mikholand – Zero Jul 19 '23 at 15:19
  • @Zero Yes, it seems to me that the headers are not being passed correctly. – mikholand Jul 19 '23 at 15:32

1 Answers1

0

Found a solution to my problem. In order to not receive a 407 response, it was necessary to receive cookies from the site and send them along with my request, which was in question.

In order not to get an error again, I used the selenium library, which launched the browser in the background and updated cookies every 5 hours.

P.S. Empirically, I came to the conclusion that cookies stop working after 8 hours.

mikholand
  • 1