How to extract html text output as list for each input from list using python web scraping. I have written code, but gives only first entry output

Question

I am new to python and programming. I am trying to extract pubchem ID from database called IMPAAT(https://cb.imsc.res.in/imppat/home). I have a list of chemical ids from the database for a herb, where going into each chemical ID hyperlink gives details on its pubchem ID and smiles data.

I have written a script in python to take each chemical ID as input and look for pubchem ID from the html page and print output to a text file using API web scraping method.

I am finding it difficult to get all the data as output. Pretty sure there is some error in the for loop as it prints only the first output many times, instead of the different output for each input.

Please help with this.

Also, I dont know how to save this kind of file where it prints input and corresponding output side by side. Please help.

import requests
import xmltodict
from pprint import pprint
import time
from bs4 import BeautifulSoup
import json
import pandas as pd
import os
from pathlib import Path
from tqdm.notebook import tqdm

cids = 'output.txt'

df = pd.read_csv(cids, sep='\t')
df

data = []

for line in df.iterrows():
    
out = requests.get(f'https://cb.imsc.res.in/imppat/Phytochemical-detailedpage-auth/CID%{line}')
    
    soup = BeautifulSoup(out.text, "html.parser")
    
    if soup.status_code == 200:
        script_data = soup.find('div', {'class': 'views-field views-field-Pubchem-id'}).find('span', {'class': 'field-content'}).find('h3')
    #print(script_data.text)
    
    for text in script_data:
        
        texts = script_data.get_text()
        
        print(text)
    
    data.append(text)
   
    
print(data)
    

****
input file consists of 

cids
0   3A155934
1   3A117235
2   3A12312921
3   3A12303662
4   3A225688
5   3A440966
6   3A443160 ```

Answer 1

There are few things you need to correct in your code.

1. Incorrect indentation of out variable.

2. Status Code should be checked on response object i.e., out not soup.

3. You don't need second loop as each response contains only single pubchem ID which you are already collecting in script_data variable.

4. Lastly, you can use pandas to associate each chemical ID to its pubchem ID and then can write to CSV file.

Refer to below code for complete result.

Code

import requests
import xmltodict
from pprint import pprint
import time
from bs4 import BeautifulSoup
import json
import pandas as pd
import os
from pathlib import Path
from tqdm.notebook import tqdm

cids = 'output.txt'

df = pd.read_csv(cids, sep='\t')

pubchem_id= []

for line in df.iterrows():
    
    out = requests.get(f'https://cb.imsc.res.in/imppat/Phytochemical-detailedpage-auth/CID%{line}')

    if out.status_code == 200:
        
        soup = BeautifulSoup(out.text, "html.parser")

        script_data = soup.find('div', {'class': 'views-field views-field-Pubchem-id'}).find('span', {'class': 'field-content'}).find('h3').getText()
    
        script_data = script_data.replace('PubChem Identifier:','')
        
        pubchem_id.append(script_data)

# As you have not mentioned column index of cids, I am assuming it should be the first column
df1 = pd.DataFrame({"chemical_id": df.iloc[:, 0].tolist(), "pubchem_id": pubchem_id})
    
print(df1)

# uncomment below line to write the dataframe into csv files & replace 'filename' by the complete filepath
# df1.to_csv('filename.csv')