2

Currently, I am using mechanize to fill up a form and send a POST request, then I am doing a regex search on the response to get the data ( a floating point number).

Is there any way I can do this by just sending a POST request? I know this is possible by using a combination of any browser's Developer Tools and the requests module to send the request but I have failed to find a comprehensive tutorial. I would also like some details about the steps involved.

SanketDG
  • 550
  • 5
  • 12

1 Answers1

2

First step: get the field name

Inspect the HTML code and find the name attribute of the field. For example, the comment form on this page is (in Chrome, right-click and choose "inspect element"):

<textarea name="comment" cols="68" rows="3" 
placeholder="Use comments to ask for more information or 
suggest improvements. Avoid answering questions in comments."
></textarea>

The field name is comment.

Step 2: assemble a dict of name: value for each field (including the hidden inputs)

Lets call it data:

data = {
   "comment": "this is a comment",
   "post_id": 1234
}

Step 3: use the data argument of `requests.post'

response = requests.post(url, data=data, cookies=cookies)

More advanced stuff

If your form has file fields you may have to check "More complicated POST requests" in the docs. Same goes for custom authentication, cookie handling, etc.

Paulo Scardine
  • 73,447
  • 11
  • 124
  • 153
  • Shouldn't the key values in `data` enclosed within double quotes? – SanketDG Jun 11 '15 at 22:14
  • Also, I have 2 urls, the url in which the form contains, and another page that get's displayed after the POST request, which request should I include here? – SanketDG Jun 11 '15 at 22:19
  • Yes, damn JavaScript habits! The URL is the one from the form's action attribute or the page URL if the form lacks this attribute (respectively your URL #2 and #1 I guess). – Paulo Scardine Jun 11 '15 at 22:25
  • I did the above mentioned steps, but I am failing to get the response. It just returns the page with the form and an additional error message ( within the html). I think I am passing the data right, but I was reading on `requests` a bit and learnt about headers...are they important? How would I define them for my usecase? – SanketDG Jun 11 '15 at 22:33
  • Ah! I found it! This is making a POST request, so it should be `requests.post(....)`. Totally missed that. – SanketDG Jun 11 '15 at 23:00
  • Sorry for that, fixed. I also don't know how I could write `requests.post` in the text and `requests.get` on the code sample. – Paulo Scardine Jun 12 '15 at 14:44