1

I attempted to scrape the Godaddy website but was unsuccessful. When I conduct a search for a particular name, it comes up "We were unable to complete your search. Please try again." and a standard browser can use it just well (without selenium). Sorry if I've inconvenienced you with this query; this is my first time doing it.

Url to scrape : https://in.godaddy.com/domainsearch/find?checkAvail=1&domainToCheck=bjmtuc.club

from selenium import webdriver


chrome_options = webdriver.ChromeOptions()
chrome_options.set_capability("goog:loggingPrefs", {"performance": "ALL", "browser": "ALL"})
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36")

driver_path = 'drive_path'
service = Service(driver_path) # service path set

driver = webdriver.Chrome(service=service, options=chrome_options) # working fine

url = 'https://in.godaddy.com/domainsearch/find?checkAvail=1&domainToCheck=bjmtuc.club'
driver.get(url)

Edit 1:

It said "Try disabling ad blockers and other extensions, enabling javascript, or using a different web browser." when I created a fake account and tried to log in. Since I don't use any extensions or ad blockers, I tried turning on javascript right away. It stated the same thing and did not work.

Code added:

chrome_options.add_argument("javascript.enabled")

Khush
  • 11
  • 2
  • Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. – Community Oct 26 '22 at 22:19

1 Answers1

1

Looks like the search doesn't work for unauthenticated users. Try opening your url manually from the incognito mode (or just logout). If you open this link as authenticated user you get the result of the search authenticated search But if you logout and then try to open this link you recieve "We were unable to complete your search. Please try again." non-authenticated search So, when you open this link manually it shows the result because you are authenticated on the site. And even if you reopen your browser and go open this link - the search result will be shown because you are still authenticated on the site. This authentication is saved into browser sessions which are stored on your computer only for your user.
But when you try to open this link using Selenium, it opens browser with its (selenium's) own saved sessions. But you didn't login into the site with Selenium, so there are no authenticated sessions. Furthermore, each time Selenium opens a browser, it creates new sessions.
So, possible solutions are:

  • Add login steps before opening the link
  • Configure selenium to use your saved sessions
  • Make a separate code for login into the site (or open the site with Selenium but login manually) and create a place where sessions for Selenium will be stored. And then configure all your tests to use this place for using saved sessions.
Eugeny Okulik
  • 1,331
  • 1
  • 5
  • 16
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 30 '22 at 08:23