2

I know google colab can be used for selenium module, but before using selenium, there should be a browser in colab virtual machine, so I have successfully installed firefox in colab by using the commands

!apt-get update
!apt install firefox

but when I try to run firefox with the command

!firefox

it throws an error

src/tcmalloc.cc:283] Attempt to free invalid pointer 0x7f4e34915040 Redirecting call to abort() to mozalloc_abort

The same problem occurs with other browsers such as chromium-browser and chrome.
I even tried 

import webbrowser
webbrowser.get('firefox').open('https://www.youtube.com')

but it throws an error saying

could not found browser location

So in overall I need the solution for this error which occurs while running a browser

Attempt to free invalid pointer 0x7f4e34915040

zx485
  • 28,498
  • 28
  • 50
  • 59
KaalKala
  • 31
  • 2
  • 4
  • @Jairath: Please don't put program names and package names in code brackets ``. Better improve the spelling and grammar and other aspects of a (soon to be) good question. – zx485 Sep 21 '19 at 20:43

2 Answers2

5

I cannot install firefox. But if you can use Chrome. Here's the code.

# install chromium, its driver, and selenium
!apt install chromium-chromedriver
!pip install selenium
# set options to be headless, ..
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
# open it, go to a website, and get results
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://www.website.com")
print(wd.page_source)  # results
# divs = wd.find_elements_by_css_selector('div')
korakot
  • 37,818
  • 16
  • 123
  • 144
  • I tried the code, but colab failed with ```WebDriverException: Message: unknown error: Chrome failed to start: crashed. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)``` – robit Nov 17 '21 at 07:43
0

These are the steps to install and run selenium in google colab with firefox driver.

  1. Install Firefox driver:
!apt-get update  
!apt install firefox
  1. Install Selenium:
!pip install selenium
  1. Import dependencies:
import selenium.webdriver as webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options as FirefoxOptions
  1. Define a function to build a scraper:
def selenium_firefox_agent(driver_path="geckodriver"):
    firefox_driver_path = f'{driver_path}'
    user_agent = 'Mozilla'
    firefox_options = FirefoxOptions()
    firefox_options.add_argument('--headless')
    firefox_options.set_preference("general.useragent.override", user_agent)

    driver = webdriver.Firefox(
        executable_path=firefox_driver_path,
        options=firefox_options
    )
    print('Scraper setup complete!')
    return driver
  1. Run the scraper:
crawler = selenium_firefox_agent()
crawler.get("https://www.google.com/")
source = crawler.page_source
Vivek Prajapati
  • 109
  • 1
  • 4