-2

I am working on a project where I have inherited some code that logs into a website using python's 'requests' library and scrapes the site for content. The 'login' code utilizes a backend URL to POST credentials to an endpoint. (Works fine)

There is another part of the site, however that does not have a backend URL. It just has a form where a pin number need to be entered to access data. Below is the div code

<input class="form-control" type="password" placeholder="Enter your
PIN number" id="pin" style="width:200px; float:left; display:inline;"
onkeypress="testPinEntry(event);">

<a id="pin_verify" style="float:left; clear:none; height:34px;" class="btn- 
glow primary login" href="#" onclick="verify_pin();">Verify PIN</a>

Does anyone know of a good way to use the requests library to input the data and press the button ? My inclination is to use the xpath and go from there (I'm pretty familiar with writing scapers - https://github.com/1jkunz1/MLB-Sabermetrics-Scraper/blob/master/src/scraper.py)

I really don't want to use selenium for this task because I don't want the project to be dependent upon it, but most of the solutions I've been able to find seem to prefer selenium.

Cœur
  • 37,241
  • 25
  • 195
  • 267
Joe
  • 512
  • 1
  • 3
  • 16

1 Answers1

2

If you're using any modern browser, you can press 'F12' to open developer tools. From there you can go to the 'Network' tab to see all requests your browsing makes.

With that tab open, send a request (a pin that you have) and a new POST request should show up in the network console. Click on it and, on the tab that opens to the left, search for "See request body" (or "edit and resend -> body" on Firefox) (or a similar name, depending on your browser) and you will be able to see all the data it sent the server as key:value pairs.

You can use Python request to send a POST request with the data you need (data={key:value, ...}). Just bear in mind some fields may need to change on each request. Some sites will haven't unique tokens valid onve for each, they will generally be either in the header in a tag or in the form itself as a hidden field.

Edit. Not the request header, what you want is the request body, sorry.

Lil Addy
  • 46
  • 4