0

I'm just using a code I found on this website https://www.bestproxyreviews.com/instagram-scraper/. The code is this:

from selenium import webdriver

    class InstagramScraper:
    
        def __init__(self, post_url):
            self.post_url = post_url
            self.comments = []
            chrome_options = webdriver.ChromeOptions()
            chrome_options.add_argument("--headless")
            self.chrome = webdriver.Chrome(chrome_options=chrome_options)
        def scrape_comments(self):
            browser = self.chrome.get(self.post_url)
            content = self.chrome.page_source
            comments = self.chrome.find_element_by_class_name("XQXOT").find_elements_by_class_name("Mr508")
            for comment in comments:
                d = comment.find_element_by_class_name("ZyFrc").find_element_by_tag_name("li").find_element_by_class_name("P9YgZ").find_element_by_tag_name("div")
                d = d.find_element_by_class_name("C4VMK")
                poster = d.find_element_by_tag_name("h3").text
                post = d.find_element_by_tag_name("span").text
                self.comments.append({
                    "poster": poster,
                    "post": post
                })
    
                return self.comments
        
    post_url = "https://www.instagram.com/p/CTu1euSp6jZ/"
    x = InstagramScraper(post_url)
    x.scrape_comments()

And the error:

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".XQXOT"}
      (Session info: headless chrome=94.0.XXXX.XX)

What I'm trying to do is to get all the comments of a post on .csv.

Any hint?

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
imatiasmb
  • 113
  • 7
  • 1
    Did you (re)search here on SO: [`[python] NoSuchElementException: Message: no such element: Unable to locate element`](https://stackoverflow.com/search?q=[python]+NoSuchElementException%3A%20Message%3A%20no%20such%20element%3A%20Unable%20to%20locate%20element) ? Then [edit] your question and tell us what these answers miss in solving your issue. – hc_dev Sep 29 '21 at 19:32

2 Answers2

1

Instagram asynchronously loads in comments and other data. To make sure they have loaded in, use the driver.implicitly_wait(30) method to ensure that Selenium has time to look for those elements, if they aren't yet loaded:

...
# Get the Instagram page
browser = self.chrome.get(self.post_url)

# Wait for comments to load
self.chrome.implicitly_wait(30) # 30 Seconds, you can lower this if your internet connection is stable and fast

# Get comments
comments = self.chrome.find_element_by_class_name("XQXOT").find_elements_by_class_name("Mr508")

# Get content now that comments have loaded
content = self.chrome.page_source
...
Xiddoc
  • 3,369
  • 3
  • 11
  • 37
  • 1
    Hope my edits fit nicely :) – Xiddoc Sep 29 '21 at 19:45
  • 1
    Helpful link to docs and explaining code-comment ️ The implicit AJAX (dynamic loading) is one of the advantages of selenium compared to manually figuring out and implementing in e.g. _requests_, _BeautifulSoup_, etc. – hc_dev Sep 29 '21 at 19:49
0

enter image description here

As you can see in the linked picture, the class for the comment button is called 'wp06b'. I searched the html of the picture for the class you're trying to use but that appears to be <ul> tags, which in html are bulleted lists. So in your code, you're trying to click on a button that doesn't exist. You could try to find the element with the xpath with the button, which would look like this

comments = WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.XPATH, '/html/body/div[7]/div[2]/div/article/div/div[2]/div/div[2]/section[1]/span[2]/button'))) 
comments.click()