0

Context: I'm trying to automatically scrape a pdf file from a website, but the file is linked to a button (pressing it starts the download). Inspecting this button with chrome developed mode informed me that the button initiates a POST request which in turn starts the file download (I assume at least). I'm trying to recreate this post request using python's requests module, but to no succes so far.

The Question: I've been googling for a couple of hours and I can't wrap my head around the header information that goes along with a POST/GET request. I can't seem to figure out why and when someone would add header information and when it is required for a successful POST request. If someone could enlighten me I'd be very grateful.

PdevG
  • 3,427
  • 15
  • 30
  • post data (e.g. form submission) goes in the request body. unless you have to set cookies or other http headers, you shouldn't have to set ANY headers to do your post request. – Marc B Sep 24 '15 at 20:04
  • How are you planning on sending the post request, if you haven't already you should check out posts like: http://stackoverflow.com/questions/4476373/simple-url-get-post-function-in-python if you are using a lib you shouldn't have to add any headers unless you need something specific sent like cookies. – scrappedcola Sep 24 '15 at 20:06
  • @ Marc B, @scrappedcola Thx for the replies. So they are only needed for special occasions? I'm sending the post request using the requests.post function from the requests module. So far it only returns a code 400 though, thought it might be the header. – PdevG Sep 24 '15 at 20:07
  • Nowadays, it's quite common for pages to include javascript code that performs ajax calls to the back-end. One simple example are login pages, where the code encrypts/manipulates the credentials before sending them over. The back-end can discriminate if the POST request is coming from an ajax call by inspecting the header named `HTTP_X_REQUESTED_WITH`, which is set by the majority of javascript frameworks – Pynchia Sep 24 '15 at 20:25

0 Answers0