Beautifulsoup find element

Question

hi i was trying to find an element i have found the first one (soup1_title) easily but i had trouble to find next one! i just need book author for example for the first one my favor output is:"by J. G. Ballard and Martin Amis" (without by) here is the issue : here is the area i need

and here is the code which i had written:

import requests
from bs4 import BeautifulSoup
#Search_Text = input('Please Enter Search Query ')
#Search_Text = Search_Text.replace(' ','+')
url = 'https://www.amazon.com/s?k=j+g+ballard+short+stories'
#print(url)
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
r = requests.get(url, headers=headers)
soup = BeautifulSoup(r.text, 'lxml')
soup1_title = soup.select('.a-color-base.a-text-normal')
soup2_title = soup.select('.a-size-base'+'a-link-normal')


for j in soup2_title:
    print(j.string)

the link is: https://www.amazon.com/s?k=j+g+ballard+short+stories

guys would you please help me in order to find the aforementioned element using Beautiful soup?

Answer 1

import requests
from bs4 import BeautifulSoup

params = {
    "k": "j g ballard short stories"
}

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:74.0) Gecko/20100101 Firefox/74.0'
}


def main(url):
    r = requests.get(url, params=params, headers=headers)
    soup = BeautifulSoup(r.content, 'lxml')
    target = soup.select("div[class$=a-color-secondary]")
    for tar in target:
        if "by" in tar.text:
            print(tar.get_text(strip=True, separator=" "))


main("https://www.amazon.com/s")

Output:

by J. G. Ballard and Martin Amis
by J. G. Ballard | Feb 1, 2010
by J. G. Ballard and Anthony Burgess
by J. G. Ballard , Ric Jerrrom , et al.
by J. G. Ballard | Sep 1, 2006
by J. G. Ballard and China Miéville
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard
by J. G. Ballard