0

I have been attempting to identify an alternative solution within Selenium 4 for extracting HTML from JavaScript. Through my research, I discovered that a method called PhantomJS, which was available in version 3.8, has since become deprecated. Despite my efforts to understand this, I consistently encounter a deprecation error, such as "AttributeError: module 'selenium.webdriver' has no attribute 'PhantomJS'". Am I overlooking something, as a significant portion of the Selenium resources I have come across employ functionalities that are no longer supported? Is it still possible to achieve this objective? I am simply seeking guidance to steer me in the right direction, and I am grateful for any advice provided. Thank you.

Fortunately, thanks to the assistance provided to me by another Stack Overflow member, I was able to identify that it will be necessary to extract the JavaScript code in order to retrieve the desired HTML content through scraping.

Attached below is the code I used to determine I need to scrape the JavaScript:

# Libraries
import urllib3
from bs4 import BeautifulSoup
from time import sleep

# Wait 3 Seconds
sleep(3)

# Store URL
url = "https://beta.goal3.xyz/sports/soccer"

# Define the headers
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/113.0.5672.127 Safari/537.11',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
    'Accept-Encoding': 'none',
    'Accept-Language': 'en-US,en;q=0.8',
    'Connection': 'keep-alive'
}

# Create an instance of the PoolManager with the specified headers
http = urllib3.PoolManager(headers=headers)

# Send a GET request to the website
response = http.request('GET', url)

# Check if the request was successful
if response.status == 200:
    # Parse the HTML content using BeautifulSoup
    soup = BeautifulSoup(response.data, 'html.parser')
    print(soup)

Here is the websites HTML: (Sorry it's hard to post in a reasonable fashion)

<!DOCTYPE html>
<html class="dark" data-theme="dark" lang="en"><title>Goal3.xyz - The first decentralized Sportsbook on zkSync Era</title><head><meta charset="utf-8"/><meta content="viewport-fit=cover width=device-width, initial-scale=1, maximum-scale=1" name="viewport"/><meta content="2" name="next-head-count"/><script async="" data-id="101408729" src="//static.getclicky.com/js"></script><noscript><p><img alt="Clicky" height="1" src="//in.getclicky.com/101408729ns.gif" width="1"/></p></noscript><meta content="yes" name="apple-mobile-web-app-capable"/><meta content="yes" name="mobile-web-app-capable"/><meta content="Experience the Future of Sport Betting with Goal3 - Powered by zkEVM. Join the GambleFi movement and unleash the power of decentralized betting!" name="description"/><meta content="sportsbook, zkSync, betting, web3, blockchain." itemprop="keywords" name="keywords"/><meta name="news_keywords"/><meta content="Goal3" property="og:site_name"/><meta content="website" property="og:type"/><meta content="https://beta.goal3.xyz" property="og:url"/><meta content="https://beta.goal3.xyz/meta/default.jpg" property="og:image"/><meta content="image/png" property="og:image:type"/><link as="style" href="/_next/static/css/f727b947152586c6.css" rel="preload"/><link data-n-g="" href="/_next/static/css/f727b947152586c6.css" rel="stylesheet"/><link as="style" href="/_next/static/css/361e7b7ce229ef67.css" rel="preload"/><link data-n-p="" href="/_next/static/css/361e7b7ce229ef67.css" rel="stylesheet"/><link as="style" href="/_next/static/css/eb36c408c8c0fcf5.css" rel="preload"/><link data-n-p="" href="/_next/static/css/eb36c408c8c0fcf5.css" rel="stylesheet"/><noscript data-n-css=""></noscript><script defer="" nomodule="" src="/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js"></script><script defer="" src="/_next/static/chunks/webpack-aa0d6a8884a8520d.js"></script><script defer="" src="/_next/static/chunks/framework-ce84985cd166733a.js"></script><script defer="" src="/_next/static/chunks/main-354042be1614f0c6.js"></script><script defer="" src="/_next/static/chunks/pages/_app-3135abef21e8aa92.js"></script><script defer="" src="/_next/static/chunks/75fc9c18-0061c64dcc709f97.js"></script><script defer="" src="/_next/static/chunks/24-67cd99fa99f141f2.js"></script><script defer="" src="/_next/static/chunks/1664-6077f61cd231cbb6.js"></script><script defer="" src="/_next/static/chunks/719-cd8014cbd98dd40e.js"></script><script defer="" src="/_next/static/chunks/6898-545aecb4c6bb194d.js"></script><script defer="" src="/_next/static/chunks/6554-1aa0abaadf7beac8.js"></script><script defer="" src="/_next/static/chunks/8208-8e5070115ebb7620.js"></script><script defer="" src="/_next/static/chunks/683-c2d3bba216c08d07.js"></script><script defer="" src="/_next/static/chunks/4159-9547181b9f859321.js"></script><script defer="" src="/_next/static/chunks/3498-3701d2045c9a344c.js"></script><script defer="" src="/_next/static/chunks/5558-29112b2bf574a06d.js"></script><script defer="" src="/_next/static/chunks/5366-90d22618471aab6b.js"></script><script defer="" src="/_next/static/chunks/pages/sports/%5Bid%5D-075d79bc27b0cb0b.js"></script><script defer="" src="/_next/static/pDbetItcMsPxqxInlsybY/_buildManifest.js"></script><script defer="" src="/_next/static/pDbetItcMsPxqxInlsybY/_ssgManifest.js"></script></head><body tabindex="-1"><script async="" crossorigin="anonymous" src="https://kit.fontawesome.com/97ee792e5e.js"></script><div id="__next"><div data-rk=""><style>[data-rk]{--rk-blurs-modalOverlay:blur(0px);--rk-fonts-body:SFRounded, ui-rounded, "SF Pro Rounded", -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";--rk-radii-actionButton:9999px;--rk-radii-connectButton:12px;--rk-radii-menuButton:12px;--rk-radii-modal:24px;--rk-radii-modalMobile:28px;--rk-colors-accentColor:#3898FF;--rk-colors-accentColorForeground:#FFF;--rk-colors-actionButtonBorder:rgba(255, 255, 255, 0.04);--rk-colors-actionButtonBorderMobile:rgba(255, 255, 255, 0.08);--rk-colors-actionButtonSecondaryBackground:rgba(255, 255, 255, 0.08);--rk-colors-closeButton:rgba(224, 232, 255, 0.6);--rk-colors-closeButtonBackground:rgba(255, 255, 255, 0.08);--rk-colors-connectButtonBackground:#1A1B1F;--rk-colors-connectButtonBackgroundError:#FF494A;--rk-colors-connectButtonInnerBackground:linear-gradient(0deg, rgba(255, 255, 255, 0.075), rgba(255, 255, 255, 0.15));--rk-colors-connectButtonText:#FFF;--rk-colors-connectButtonTextError:#FFF;--rk-colors-connectionIndicator:#30E000;--rk-colors-downloadBottomCardBackground:linear-gradient(126deg, rgba(0, 0, 0, 0) 9.49%, rgba(120, 120, 120, 0.2) 71.04%), #1A1B1F;--rk-colors-downloadTopCardBackground:linear-gradient(126deg, rgba(120, 120, 120, 0.2) 9.49%, rgba(0, 0, 0, 0) 71.04%), #1A1B1F;--rk-colors-error:#FF494A;--rk-colors-generalBorder:rgba(255, 255, 255, 0.08);--rk-colors-generalBorderDim:rgba(255, 255, 255, 0.04);--rk-colors-menuItemBackground:rgba(224, 232, 255, 0.1);--rk-colors-modalBackdrop:rgba(0, 0, 0, 0.5);--rk-colors-modalBackground:#1A1B1F;--rk-colors-modalBorder:rgba(255, 255, 255, 0.08);--rk-colors-modalText:#FFF;--rk-colors-modalTextDim:rgba(224, 232, 255, 0.3);--rk-colors-modalTextSecondary:rgba(255, 255, 255, 0.6);--rk-colors-profileAction:rgba(224, 232, 255, 0.1);--rk-colors-profileActionHover:rgba(224, 232, 255, 0.2);--rk-colors-profileForeground:rgba(224, 232, 255, 0.05);--rk-colors-selectedOptionBorder:rgba(224, 232, 255, 0.1);--rk-colors-standby:#FFD641;--rk-shadows-connectButton:0px 4px 12px rgba(0, 0, 0, 0.1);--rk-shadows-dialog:0px 8px 32px rgba(0, 0, 0, 0.32);--rk-shadows-profileDetailsAction:0px 2px 6px rgba(37, 41, 46, 0.04);--rk-shadows-selectedOption:0px 2px 6px rgba(0, 0, 0, 0.24);--rk-shadows-selectedWallet:0px 2px 6px rgba(0, 0, 0, 0.24);--rk-shadows-walletLogo:0px 2px 16px rgba(0, 0, 0, 0.16);}</style></div></div><script id="__NEXT_DATA__" type="application/json">{"props":{"pageProps":{}},"page":"/sports/[id]","query":{},"buildId":"pDbetItcMsPxqxInlsybY","nextExport":true,"autoExport":true,"isFallback":false,"locale":"en","locales":["en"],"defaultLocale":"en","scriptLoader":[]}</script></body></html>
GildedWasp
  • 1

0 Answers0