4

I'm currently using AWS Apache Airflow service (MWAA) and I've been needing to run some web scraping code using selenium, I did manage to add "chromedriver" and "selenium" to the dependencies of the project, but I've been struggling to add chromium to it (or any headless web browser).

AWS only allows a requirements.txt as dependencies to be installed using pip for MWAA, therefore I can't just add a "sudo yum install -y chromium" to the building cycle. I would like to know if it would be possible to add a chromedriver to the environment.

Thanks in advance!

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Lucas
  • 161
  • 1
  • 8

1 Answers1

0

You can include the webdriver-manager in your requirements.txt as described here. This package will automate the download and setup of your chromium driver.

Then you can use selenium with chromium:

# selenium 4
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromiumService
from webdriver_manager.chrome import ChromeDriverManager
from webdriver_manager.core.utils import ChromeType

driver = webdriver.Chrome(service=ChromiumService(ChromeDriverManager(chrome_type=ChromeType.CHROMIUM).install()))
DavidBu
  • 478
  • 4
  • 6