0

I am trying to scrape website with Beautiful Soup. After printing the container, It gives me an empty list. How can I fix this?

import requests
from bs4 import BeautifulSoup
import lxml

URL = 'https://www.monster.com/jobs/search/?q=Software-Developer&where=Australia'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'lxml')
container = soup.find_all('div', class_="results-list")
print(container)
Zabih Ullah
  • 43
  • 1
  • 4
  • 1
    The page data you received doesn't include a div containing 'results-list': `container = soup.find_all('div')` `'div' in str(container) # -> true (sanity check)` `'results-list' in str(container) # -> false` `'class="results-list' in str(soup) # -> false` However, it is in the result you receive, but only as a style: `'results-list' in str(soup) # -> true` `'.results-list' in str(soup) # -> true` – Boom100100 Jun 18 '21 at 15:34

1 Answers1

1

The data is loaded from external URL via Ajax call. You can use requests module to simulate it and load the data:

import json
import requests
from textwrap import wrap
from bs4 import BeautifulSoup


api_url = "https://services.monster.io/jobs-svx-service/v2/monster/search-jobs/samsearch/en-us"

payload = {
    "fingerprintId": "",
    "jobAdsRequest": {
        "placement": {"appName": "monster", "component": "JSR_SPLIT_VIEW"},
        "position": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
    },
    "jobQuery": {
        "companyDisplayNames": [],
        "excludeJobs": [],
        "locations": [{"address": "Australia", "country": "us"}],
        "query": "Software-Developer",
    },
    "offset": 0,
    "pageSize": 20,
    "searchId": "",
}


data = requests.post(api_url, json=payload).json()

# uncomment to print all data:
# print(json.dumps(data, indent=4))

for r in data["jobResults"]:
    s = BeautifulSoup(r["jobPosting"]["description"], "html.parser")
    print("\n".join(wrap(s.get_text(strip=True, separator=" "))))
    print("-" * 80)

Prints:

Description: The role of the Mobile Developer is to analyze business
requirements, develop a design plan, translate plan into program
specifications (or a manual process), code, test, and coordinate
implementation. They will be focusing on enhancements within existing
applications (not building out anything new). Working with multiple
applications- all very client specific. This developer could also be
tasked with some frontend work for a specific application. Need to be
able to utilize Angular skills for frontend development. -Must be
highly proficient with mobile platform Application Programming
Interfaces (API) such as Apple iOS and Android Mobile -Hands on
experience with latest iOS and Android tech -Deep knowledge of Angular


...and so on
Andrej Kesely
  • 168,389
  • 15
  • 48
  • 91