1

im getting started with web scraping the page loads more data when it scrolls so i opened the get url when it does this and got a json of all the data i needed but my code doesnt return all the json i get in the browser only 2 results that arent the first or last 2 or even next to eachother i have no idea why its cutting the rest out

import requests

jsonUrl = "https://www.deviantart.com/_napi/da-user-profile/api/gallery/contents?username=Dreamsquid&offset=48&limit=24&folderid=58533887"
response = requests.get(jsonUrl)
jsonResponse = response.json()
print(jsonResponse)
j. doe
  • 21
  • 3

1 Answers1

1

Tried running the same python commands on my console and compared the console results with the result that we are getting from opening the url in chrome. Then I formatted and compared the complete json on an online json comparer (Json-Diff) but I can't really see any difference (except the " and ' difference)

I think the problem might be because of your console. As the json is a bit big, it might not be able to print the complete json. I would suggest to write the output to a file and then compare the results

Code for the same :-

import requests

jsonUrl = "https://www.deviantart.com/_napi/da-user-profile/api/gallery/contents?username=Dreamsquid&offset=48&limit=24&folderid=58533887"
response = requests.get(jsonUrl)
jsonResponse = response.json()
print(jsonResponse)
with open('data.json', 'w') as json_file:
    json.dump(jsonResponse, json_file)

Hopefully, this might answer your question. If this doesn't work or any other issue or anything, please do let me know in the comments section.

Thanks

Pratap Alok Raj
  • 1,098
  • 10
  • 19
  • Hey J.Doe, just some tips and extra information for your future web-scrapping projects. If the web-page is filling in the data using JavaScript you'll have to run it through a real browser, such as Chrome Headless using the Selenium library. Please refer the below post for reference: - https://stackoverflow.com/questions/47730671/python-3-using-requests-does-not-get-the-full-content-of-a-web-page/47730866 – Pratap Alok Raj May 16 '21 at 17:31