Why the response of httpx is different compared to inspect element network?

Question

I am trying to scrape this site https://shopee.co.id/S-B-Golden-Curry-Bumbu-Kari-Japanese-Curry-Mix-Medium-Hot-220-gr-i.65323877.3343602079?sp_atk=0b471505-cfca-491b-b7a2-453d455eecf6&xptdk=0b471505-cfca-491b-b7a2-453d455eecf6

I am using python httpx and the below is my code

url=f"https://shopee.co.id/api/v4/item/get?shopid=65323877&itemid=3343602079"
headers = { "User-Agent" : "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"
            }
resp=httpx.get(url,headers=headers)

The returned json is as follows:

{'error': None,
 'error_msg': None,
 'data': {'itemid': 3343602079,
  'shopid': 65323877,
  'userid': 0,
  'price_max_before_discount': -1,
  'has_lowest_price_guarantee': False,
  'price_before_discount': 0,
  'price_min_before_discount': -1,
  'exclusive_price_info': None,
  'hidden_price_display': None,
  'price_min': 9315023372,
  'price_max': 9315023372,
  'price': 9315023372, ...

I see that the price shown in the json is different than the one on the web;7490000000. Can someone please help?

score 0 · Accepted Answer · answered Jun 22 '23 at 17:46

Try to send a request without User-Agent header. It seems that the server sends different content based on this information:

import requests

url = 'https://shopee.co.id/api/v4/item/get?shopid=65323877&itemid=3343602079'

data = requests.get(url).json()
print(data)

Prints:

{
    "error": None,
    "error_msg": None,
    "data": {
        "itemid": 3343602079,
        "shopid": 65323877,
        "userid": 0,
        "price_max_before_discount": -1,
        "has_lowest_price_guarantee": False,
        "price_before_discount": 0,
        "price_min_before_discount": -1,
        "exclusive_price_info": None,
        "hidden_price_display": None,
        "price_min": 7490000000,
        "price_max": 7490000000,
        "price": 7490000000,
        "stock": 0,

...

Why the response of httpx is different compared to inspect element network?

1 Answers1