0

Please refer the code :

```soup=BeautifulSoup(browser.page_source, "html.parser")
rlist=soup.find_all('div',{"class": "WMbnJf gws-localreviews__google-review"})# whole orange box class="WMbnJf gws-localreviews__google-review <REVIEW ID GOES HERE>

print('I REACHED HERE')
try:
    print('rlist type: ',type(rlist))
except:
    print('rlist:\n',rlist)
    #result=[]

for r in rlist:
    r1=r.find('div',{"class": "jxjCjc"})
    print('\n','r1 type: ',type(r1),'\n\n') ##

    r2=r1.find('div',{"class": 'TSUbDb'})
    personname=r2.text
    print(personname) ##
    try:
        r3=r1.find('div',{"class": 'Jtu6Td'})
        personreview=r3.text
    except:
        personreview='-' #Blank Review

    print(personreview)

    cur.execute('''INSERT INTO Reviews (placeid, name, review)
    VALUES ( ?, ?, ? )''', (pid, personname, personreview) )
    conn.commit()```

I ran the code several times to find that the whole for-loop isn't working. Can't figure this out on my own, please help!!

Here's the code before the beautiful soup parser:

browser.get(placeidurl)                        

wait = WebDriverWait(browser, 10)
menu_bt = wait.until(EC.element_to_be_clickable(
                       (By.XPATH, '//g-dropdown-button[@class=\'dkSGpd NkCsjc\']'))
                   )
menu_bt.click()

recent_rating_bt=browser.find_element_by_xpath("//g-menu[@role='menu']//div[@class='znKVS'][text()='Newest']")

recent_rating_bt.click()
time.sleep(5)```
Ishika Johari
  • 1
  • 1
  • you'll have to supply more code than that, can't do much with what you provided other than what I'm seeing is that the page is likely dynamic. The classes you are looking for probably are rendered/change each time, thus when you use bs4 to find it, it's not there – chitown88 May 08 '20 at 15:36
  • @chitown88 Here i have used Selenium and beautiful soup. Correct me if I'm wrong but as far as i know, both work fine with dynamic sites as well, don't they? Otherwise could you suggest me something else? Thankyou in advance – Ishika Johari May 08 '20 at 16:55
  • They are 2 different things. Selenium is a browser simulator. It’ll create a process that simulates the actions of opening a browser and “clicking” around. You CAN also parse the html with selenium. Beautifulsoup is solely to parse the html. You’ll need to get the html response first. – chitown88 May 08 '20 at 21:35
  • So beautifulsoup doesn’t work with dynamic sites in the sense that it doesn’t know the difference. It just understands html, and you need something else (like selenium or requests) to fetch the html source code for a site. – chitown88 May 08 '20 at 21:37
  • So what I mean by saying you need to give more info, is I need to see how your getting the html. So the part of the code before you create the beautifulsoup object. – chitown88 May 08 '20 at 21:39
  • @chitown88 I've added the code, sorry for the inconvenience – Ishika Johari May 09 '20 at 04:56
  • No inconvenience, there’s just not much more one can do if can’t see the whole process. I’ll take a look as soon as I can. – chitown88 May 09 '20 at 07:38
  • And now looking, you still haven’t provided it all. I can’t run the code without placeidurl defined. I can’t run cur.execute, etc. please provide enough code to run on our ends so that we can give a working solution. – chitown88 May 09 '20 at 07:47
  • I know you’re new, but conventional practice on SO is: describe the problem (IE. tells us what you are trying to do, what isn’t working, and what the output should look like). Then give enough of the code so that we can reproduce that on our end. It’s like taking your car in to get checked, telling the mechanic it’s not running correctly, but then not giving them the keys to start the engine or even letting the mechanic look under the hood. There’s only so much the mechanic can determine if they can’t debug/troubleshoot. – chitown88 May 09 '20 at 07:53

0 Answers0