0

I am currently using python to web scrape the three-point statistics for every NBA player and am trying to put this data in a data frame. The code below is my attempt at adding the values to the data frame. The variables players,teams,threePointAttempts, and threePointPercentage are all lists containing 50 values. These are refilled after every iteration of the while loop because the script moves through each page of the NBA site.

while i<10:
soup = BeautifulSoup(d.page_source, 'html.parser').find('table')
headers, [_, *data] = [i.text for i in soup.find_all('th')], [[i.text for i in b.find_all('td')] for b in soup.find_all('tr')]
final_data = [i for i in data if len(i) > 1]

data_attrs = [dict(zip(headers, i)) for i in final_data]
print(data_attrs)

players = [i['PLAYER'] for i in data_attrs]
teams = [i['TEAM'] for i in data_attrs]
threePointAttempts = [i['3PA'] for i in data_attrs]
threePointPercentage = [i['3P%'] for i in data_attrs]


data_df = data_df.append(pd.DataFrame(players, columns=['Player']),ignore_index=True)
data_df = data_df.append(pd.DataFrame(teams, columns=['Team']),ignore_index=True)
data_df = data_df.append(pd.DataFrame(threePointAttempts, columns=['3PA']),ignore_index=True)
data_df = data_df.append(pd.DataFrame(threePointPercentage, columns=['3P%']),ignore_index=True)
data_df = data_df[['Player','Team','3PA','3P%']]

The issue I am having is the data frame fills like this:

First columnSecond columnThird column

Community
  • 1
  • 1
Brennan Mosher
  • 17
  • 2
  • 5

1 Answers1

1

Try:

temp_df = pd.DataFrame({'Player': players,
                        'Team': teams,
                        '3PA': threePointAttempts,
                        '3P%': threePointPercentage})

data_df = data_df.append(temp_df, ignore_index=True)
Luke Ning
  • 147
  • 6
  • 1
    Thank you for the response. The code you posted has an error in it. It should be data_df = data_df.append(temp_df, ignore_index=True). Otherwise, it worked perfectly for me. – Brennan Mosher Mar 08 '19 at 20:54
  • @BrennanMosher you're right, and I'm glad I could help :) – Luke Ning Mar 08 '19 at 21:24
  • @BrennanMosher, welcome to SO. make sure you accept the answers when a working/appropriate solution is posted to your question. – chitown88 Mar 21 '19 at 15:01