1

I am having trouble using explicit wait with Selenium. I keep getting the following error.

File "C:/Users/username/path/craigslist_scrape.py", line 108, in <module>
    WebDriverWait(driver,5).until(EC.text_to_be_present_in_element_value(By.XPATH('//p[@class="reply-email-address"]/a[@class="mail app"]')))

TypeError: 'str' object is not callable

My code is as follows. Basically, I am trying to scrape the email address. However, most of the time, the email address is not visible unless I click 'reply'. What I am trying to do is to get Selenium to click Reply and wait until the email address is visible and then scrape. I have tried using a couple of different methods, but kept getting the same error. Not sure what exactly happened. Hope somebody can help. 

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.select import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import TimeoutException

driver=webdriver.Chrome()
driver.get('https://nh.craigslist.org/search/reb')

link_list=[]

link=driver.find_elements_by_xpath('//li[@class="result-row"]/a[@href]')
for elem in link:
    print(elem.get_attribute('href'))
    link_list.append(elem.get_attribute('href'))
for link in link_list:
    driver.get(link)
    driver.find_element_by_xpath('//div[@class="actions-combo"]/button[@role="button"]').click()
    WebDriverWait(driver,5).until(EC.text_to_be_present_in_element_value(By.XPATH('//p[@class="reply-email-address"]/a[@class="mail app"]')))
    email=driver.find_element_by_xpath('//p[@class="reply-email-address"]/a').text
    print(email)
Fatchoco
  • 31
  • 1
  • 6
  • I think you should be more specific, where this error appeared ... because I would have to run the code to find out, what could went wrong ... Please provide a full error log :) – StyleZ Apr 20 '20 at 00:39
  • It's with the WebDriverWait function. Can't get it to work. And how do you get the full error log? Sorry a bit new to coding. – Fatchoco Apr 20 '20 at 01:05
  • no at all :) When it gives an error, just copy paste it, I am sure that its not just the 1 line error :D – StyleZ Apr 20 '20 at 01:06
  • WebDriverWait(driver,5).until(EC.text_to_be_present_in_element_value(By.XPATH('//p[@class="reply-email-address"]/a[@class="mail app"]'))) TypeError: 'str' object is not callable – Fatchoco Apr 20 '20 at 01:12
  • This is where it throws the error. I have tried a variety of ECs, but can't get it to work. Basically I am trying to make sure that the element is present before I can scrape it. – Fatchoco Apr 20 '20 at 01:13

1 Answers1

2

Okay I have realised what you want to do ...

The problem is that you are calling the wrong function with wrong parameters. Here is the correct way to call it :)

WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, '/html/body/section/section/header/div[2]/div/div[1]/aside')))

The main problem is that you are telling the function that which element it should be looking for using which function ...

Here is SO answer that answers it better than I do: Selenium - wait until element is present, visible and interactable

StyleZ
  • 1,276
  • 3
  • 11
  • 27
  • This works. Thank you! Just wondering if scraping not visible elements with BeautifulSoup is faster than with Selenium? – Fatchoco Apr 20 '20 at 02:15
  • I am not sure, but if you care about speed, I think you chose a wrong language for the job :D ... also, please accept the answer – StyleZ Apr 20 '20 at 07:11