from bs4 import BeautifulSoup
import requests
import html5lib
url = 'https://twitter.com/st3phensparkman'
result = requests.get(url)
doc = BeautifulSoup(result.text, 'html5lib')
followers = doc.find_all(text='Followers')
print(followers)
For some insight - I've been coding more and more recently (new to the game), however I've become stumped when dealing with web scraping. After countless tries, I managed to build one - now though, I'm aiming to create my own (without having to copy a YouTubers code). My project will aim to find how many followers my friends and myself have on twitter.
I've been using BeautifulSoup, requests, and of course, the built in html parser.
I didn't get too far before the problem arises. When I'm trying to locate the first tag/string. The program runs successfully, however, it only returns empty brackets. Searching for answers, I've found that it could be the parser, people online have said that it's not built for all html doc's that are used by big websites.
A substitute parser, html5lib, is supposed to work. Upon downloading it though, my program runs the error that it can't find that module!
A solution to either of these problems should set me on the right track. Is there a way to make it return the true value (not empty brackets)? Or should I use html5lib, and if so, why can't my computer find the module?
FYI - I'm running the program inside a venv. While I suspect this is the issue using html5lib, by checking in the console, I've concluded that it is indeed downloaded and updated
I tried using html5lib in order to make the tags/strings appear in brackets Created a new problem; "Cannot find module"
P.S I can't include screenshots, so I pasted the html5lib code to the top. My other bit of code is the exact same, except it doesn't contain "import html5lib" and I replaced the parser accordingly
