How can I search for a HTML element using Python? Python Web Bot Error

Question

I'm following a tutorial to create a bot that can perform tasks on any web page. I am using Python3 to search any random website, then using the search results (from that site) to print data. I have imported a selenium webdriver, and have ensured that it is set up correctly.

The problem i'm facing is i'm trying to create a for loop that cycles through search results. This for loop is using the class name from the website i'm testing - so the bot can identify article elements. The issue is the class name is: c-entry-box--compact__title This is causing SyntaxError: Cannot assign to literal

Is there any way around this? This websites' search results doesn't have any other shorter class names or ids that are shorter, nor does it contain hyphens or underscores. I am running my code on test website 'theverge's' search results.

Relevant code:

try:
    main = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "c-entry-box--compact__body"))
    )

    articles = main.find_element_by_class_name("c-entry-box--compact__title")

    for "c-entry-box--compact__title" in articles:
        header = articles.find_element_by_class_name("c-entry-box--compact__title")
        print(header.text)
finally:
    driver.quit()

Any tips or ideas to point me in the right direction is greatly appreciated!

Update: 11:44pm 21/8

I created a variable for the class name. Now the error is

...line 28, in <module>
for article in articles:
TypeError: 'WebElement' object is not iterable

Update 12:12am 22/8

I made the recent posters changes and tweaked some of my code. The only error i get now is to do with the use of keyboard entries, or keys. It is a AttributeError: 'list' object has no attribute 'send_keys' My code is

search_button = driver.find_elements_by_id("icon-search")
search = driver.find_elements_by_name("q")
search.send_keys('facebook')
search.send_keys(Keys.RETURN)

Answer 1

To get all header text Induce WebDriverWait() and wait for visibility_of_all_elements_located() and following css selector.

driver.get("https://www.theverge.com/")
headerelements=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.c-entry-box--compact__body>h2>a")))
for head in headerelements:
    print(head.text)

Console Output:

BRYDGE’S LATEST KEYBOARDS TURN A SURFACE PRO OR GO INTO A STANDARD LAPTOP
Ikea gives its 2021 catalog an Animal Crossing-themed makeover in Taiwan
School nurses are on the frontlines of the pandemic
AN INNOCENT TYPO LED TO A GIANT 212-STORY OBELISK IN MICROSOFT FLIGHT SIMULATOR
The epic campaign to win Elon Musk’s Tesla factory with memes
NASA is going to try to hunt down a leak on the International Space Station this weekend
What’s the best student laptop? We asked students
Goodbye to Patriot Act, a comedy show that was a different kind of angry
How to pick the right headphones for kids
Swipe left, Elon stans: that Tesla dating app is a joke, for now
Leaked Google Pixel 5 renders show dual rear camera and fingerprint sensor
Minecraft Education is perfectly suited for this surreal back-to-school moment
What we listen to while working from home
Samsung’s Galaxy S20 is receiving Note 20 features with new One UI update
Facebook’s old web design will disappear in September
Apple reportedly using cheaper iPhone battery parts to offset 5G cost
THE VERGE’S BACK TO SCHOOL SPECIAL
Epic to host a #FreeFortnite tournament with anti-Apple prizes
After inking a deal with Netflix, Trump impersonator Sarah Cooper is also getting a TV show
Magic Leap’s lost work The Last Light gets a surprise release after its developers were laid off
Android 11 phones will summon Android Auto wirelessly, no need to pull out your device
HOW FORTNITE’S EPIC BATTLE WITH APPLE COULD RESHAPE THE ANTITRUST FIGHT
Adobe accidentally deleted people’s photos in latest Lightroom update
Major news publishers ask Apple what can get them an App Store deal like Amazon’s
Tesla is working on a sensor that can detect a child left behind in a hot car
Fertility app Premom reportedly shared customer data with Chinese companies
Mark Zuckerberg testified before the FTC as part of its Facebook antitrust probe
How to get Microsoft’s xCloud and stream Xbox games on your phone right now
Where to sit on the school bus just got a lot more complicated
Former Uber security chief charged with paying hush money to cover up 2016 hack
Google confirms Android 11 will limit third-party camera apps because of location spying fears
Uber and Lyft shutdown in California averted as judge grants emergency stay
Netflix is re-creating iconic Stranger Things sets in LA, and you can drive your car through them
Google’s Pixel Buds are now available in more colors nearly four months after launch
Airbnb puts global ban on house parties to support social distancing guidelines
HOUSES ARE INFLUENCERS NOW, AND THIS ONE BURNED TO THE GROUND
Lyft will suspend its ride-hailing service in California
Reddit reports 18 percent reduction in hateful content after banning nearly 7,000 subreddits
A mail-in COVID-19 test company switched to FedEx because of USPS delays
Steve Bannon charged with fraud over crowdfunded border wall
Razer gets into the ergonomic game with its new $99.99 Pro Click wireless mouse
SAMSUNG GALAXY NOTE 20 ULTRA REVIEW: BIG PHONE, SMALL UPDATES
Google’s Pixel Buds get new transcribe mode, attention alerts, and sharing detection
Control’s publisher explains why it won’t offer a free next-gen upgrade
SpaceX still pressing ahead with its Air Force lawsuit, despite winning coveted Air Force contract
We're building great things, and we need your talent.
DoorDash launches grocery delivery to compete with Amazon and Instacart

---

For your script there is problem articles = main.find_element_by_class_name("c-entry-box--compact__title")

find_element_by_class_name() will return single webelement. To get list of elements you need to use find_elements_by_class_name()

Therefor it should be

articles = main.find_elements_by_class_name("c-entry-box--compact__title")

However I would suggest use my approach which is very linear.