12

Currently I have Selenium hooked up to python to scrape a webpage. I found out that the page actually pulls data from a JSON API, and I can get a JSON response as long as I'm logged in to the page.

However, my approach of getting that response into python seems a bit junky; I select text enclosed in <pre> tags and use python's json package to parse the data like so:

import json
from selenium import webdriver

url = 'http://jsonplaceholder.typicode.com/posts/1'
driver = webdriver.Chrome()
driver.get(url)
json_text = driver.find_element_by_css_selector('pre').get_attribute('innerText')
json_response = json.loads(json_text)

The only reason I need to select within <pre> tags at all is because when JSON appears in Chrome, it comes formatted like this:

<html>
<head></head>
<body>
<pre style="word-wrap: break-word; white-space: pre-wrap;">{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}</pre>
</body>
</html>

And the only reason I need to do this inside selenium at all is because I need to be logged into the website in order to get a response. Otherwise I get a 401 and no data.

Roger Filmyer
  • 676
  • 1
  • 8
  • 24

1 Answers1

21

You can find the pre element and get it's text, then load it via json.loads():

import json 

pre = driver.find_element_by_tag_name("pre").text
data = json.loads(pre)
print(data)

Also, if this does not work as-is, and, as suggested by @Skandix in comments, prepend view-source: to your url.

Also, you may avoid using selenium to get the desired JSON data and transfer the cookies from selenium to requests to keep "staying logged in", see:

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
  • 1. Good to know that the `
    ` method works (although `.text` is definitely nicer than my approach `.get_attribute('innerText')`)...
     – Roger Filmyer May 09 '16 at 17:45
  • 2. TIL about `requests`'s cookies support. I managed to avoid another page load of this 7MB angular app from hell, shave a few seconds off my script run time, and write cleaner code. Thanks! – Roger Filmyer May 09 '16 at 17:47
  • 5
    for those of you who have trouble with firefox autoformating the json-response into a table and changing the source: add **"view-source:"** in front of your url. Then you can continue using find-by-tag "pre". – Skandix Jul 13 '18 at 13:19
  • 2023 use: driver.find_element(By.TAG_NAME, 'pre').text – Chris Jun 22 '23 at 15:36