2

I am learning web crawler

I use postman as a tool

Currently practicing on the shopee website

I try to intercept "add to cart" and "checkout" posts

and send the complete duplicate post in one go

At first it was smooth

Every request I make will be responded by the server

But the next day the server suddenly stopped responding to my post

return data

{
    "tracking_id": "cc86ce23-1e8f-49f1-b304-752fb6cc7ead",
    "action_type": 2,
    "error": 90309999,
    "is_customized": false
}

I try to manually checkout again and fetch the post

The code has not been updated or changed

within tens of seconds after manual checkout

The post I send can be responded to by the server

But after dozens of seconds, the server will no longer respond to this post

I compared the difference between "can be responded to" and "can not be responded to" post

I found "cannot be responded to" post, his cookie has been changed

but i don't know how to solve this problem

The "Add to Cart" post is now valid

However, the "checkout" post is only valid within tens of seconds after the manual order is placed

I cleared the cookies and even changed to a new computer and ip, but I can't solve this problem

How can I solve this problem?

Marco Bonelli
  • 63,369
  • 21
  • 118
  • 128
可兒本舖
  • 21
  • 1
  • 3
  • I am able to use cURL command (for Windows CMD) generated from Chome developer console and get the API result successfully. But the cURL for Linux shell would fail. Are you able to repeat this? I am currently investigating this behaviour. – AlexanderK1987 Nov 04 '22 at 03:48
  • Update: curl 7.86.0 is able to fetch data with header 'af-ac-enc-dat: null'. But curl 7.85.0 cannot – AlexanderK1987 Nov 04 '22 at 13:24

3 Answers3

1

You can add a requests header, like this headers key af-ac-enc-dat value null, this works for me.

Zach Jensz
  • 3,650
  • 5
  • 15
  • 30
Pans
  • 19
  • 2
  • This work for me. But somehow, when I try to get detailed products, it returns different price values. This's the URL that I tried: ```https://shopee.co.id/api/v4/item/get?itemid=000000&shopid=000000``` – Felix gani harris Oct 24 '22 at 09:13
  • I tried but, it didn't work. Getting {"code":10011,"msg":"err:invalid request, detail:Illegal request"} response – kumardippu Mar 23 '23 at 05:06
0

Most sites have updated the conditions for checking headers. According to XHR breakpoint '/search_ items?‘ You can see the contents of the check.
’X-Sap-Access-S': It is encrypted from md5 and encoded with base64.Intuition tells me it's like a timestamp, I'm not sure.
’X-Sap-Access-T ': It is an int (time. time()) timestamp. I have verified it.
‘X-Sap-Access-F’：
‘af-ac-enc-dat’：
These two are difficult. I don't know how they were generated...

headers = {
‘X-Sap-Access-F’: "3.0.0.6.0|13|2.4.12_5.1.0_0_187|d7fdb0e639214ead93fc55232d583a5672d3fc79a4974e|10900|1100"，    
‘X-Sap-Access-S’: "MUZALjKcrgAuyFpkJuSsYBpwPHzjuPHbPVu1IfSJeJA="，    
‘X-Sap-Access-T’: "1667569629"，    
‘af-ac-enc-dat’: "AAcyLjQuMS0yAAABhELlOE8AAAtvAkAAAAAAAAAAAOYyhFAbVQMMpIKa2+dGIBkKaWUVkWOzjLDykZY2dhCO2aemlognVlm0dDMGivrgRXrrMUseuXD/py3eUhWKg7iganQiVFma6ooH+ehDhBTg74+38rWtzNgtemYl8gaHOc5BdJupajpcNcL9Yg+yzg0eZ4Bzj+zJuu1NkJs+QlLrpYU6CYdLxz8yJkk2EqrDfkG48jSZUbatRPTf42k24UXHBZsir4fwWxQLVKw5gDu5CyybpSVWFCd7OLsY30Hj1OjSKZvDNTpkAhYqvdOyLCTGr41kHyFGV3ZaoQ01NX1u6R9AnBG6X9s1ynZK6vnTBgzBIOKTNFS4j1VT8sOl1BEtObri8ZUW3OTOHeCO4vGDCq4gRJFmvwSm1BNdccjxAekgEx3xwroP6ZL6LO5bh9QSxuKGYkUmR84CcHLB6dmMPnXDUGkagca9MFiK8RmRsrN2vcLDNTpkAhYqvdOyLCTGr41kjULvHyB47grB1wJBPIqJXX85i0ATioYJG+gPR+8Iz8TP3jANhks7Nl1goVRGmInlF+8hpeeDkMDl0WXkPHJxWfUHoucLt/ap9WR6Bk8psQ0JvTBAKC2yDgEHlbMyYBASMCWwUsMjN91gY0M/3PoYDZwkdZaPXbTOQWIv6XoNZwJXoG/G+6MnZ1XOfnl6yuBbpKOPnEICh33J8hrs9Z7IhC8PysgyvIxGQ0MuU2DTK0+RY7OMsPKRljZ2EI7Zp6aWkWOzjLDykZY2dhCO2aemlkmIKiMicLiHq6lIrEXcEhaLGHvdZJ2TwxpZ36RHEhSc"    }
Will
  • 1
0

curl 7.86.0 with set af-ac as null seems no longer work for single product detail scraper. But scraper products search page still working so far.

HSSFOX
  • 1