html table scraping in python

Question

I wanted to scrape an html table with this code

import requests
from bs4 import BeautifulSoup

page1 = requests.get("http://kworb.net/spotify/country/br_weekly.html")                                                  
soup = BeautifulSoup(page1.content, 'html.parser')
for tr in soup.findAll('tr'):
    tds =tr.find_all('td')
    print(tds[0].text)

it seems to work: I'm able to get the table and each of its rows in a different tds list. Except that when I try to get the first column for each row (tds[0].text) there's an error.

Could you provide some clues?

what error ? always put full error message (Traceback) in question (as text, not screenshot). There are other useful informations. — furas, Jan 08 '18 at 01:20
use `print()` to see what you have in `tds`. You may get row with `` instead of `` and then you get empty `tds` — furas, Jan 08 '18 at 01:22
no, tds isn't empty example: [200, -1,
MC Gudan - Boca de Pêlo
, 9, 144, , 350,693, -2,742, 3,455,285] — , Jan 08 '18 at 01:26
I get empty `tds` for first `tr` because there are `` with headers — furas, Jan 08 '18 at 01:27
Are you expecting more answers to come? I think you have already got your requirement satisfied with either of the solutions available below. Pick the one you like most and accept that as an answer so that people can skip your thread and go for solving another @Pablo Castaño. — SIM, Jan 08 '18 at 10:33

furas · Accepted Answer · 2018-01-08T09:34:08.213

1

First row has headers <th> instead of <td> so you get empty tds - and you have to check size for tds

if len(tds) > 0:
    print(tds[0].text)

or shorter

if tds:
    print(tds[0].text)

Or you can skip first row using [1:]

for tr in soup.find_all('tr')[1:]:
   tds = tr.find_all('td')
   print(tds[0].text)

edited Jan 08 '18 at 09:34

answered Jan 08 '18 at 01:26

furas

134,197
12
106
148

Thks, first row was empty! – Jan 08 '18 at 01:30

score 0 · Answer 2 · answered Jan 08 '18 at 01:31

Try to find the tbody first, then find all tr under it, as all data are in table body, you have to skip look in the table head:

soup = BeautifulSoup(page1.content, 'html.parser')
tbody = soup.find("tbody")
for tr in tbody.findAll('tr'):
    tds =tr.find_all('td')
    print(tds[0].text)

score 0 · Answer 3 · answered Jan 08 '18 at 01:49

You can try this:

import urllib
from bs4 import BeautifulSoup as soup
final_data = [i.text for i in soup(str(urllib.urlopen('http://kworb.net/spotify/country/br_weekly.html').read()), 'lxml').find_all('tr')]

Output:

[u'PosP+Artist and TitleWksPk(x?)StreamsStreams+Total', u'1=Anitta - Vai malandra (Participa\xe7\xe3o especial Tropkillaz e DJ Yuri Martins)31(x3)7,324,508-2,113,00224,479,067', u'2+1MC Jhowzinho e MC Kadinho - Agora Vai Sentar171(x3)3,264,965-147,87633,465,852', u'3-1Anitta - Downtown71(x4)3,139,316-434,81623,888,009', u'4=Mc Don Juan - Amar Amei93(x4)2,883,304-174,49721,564,797', u'5=Matheus & Kauan - Deixa Ela Beijar742,654,291-48,60616,602,124', u'6=Mc Kevinho - Rabiola562,648,786+40,21110,561,248', u'7=Lucas Lucco - Permanecer952,399,311-200,82618,285,973', u'8=Wesley Safad\xe3o - Ar-Condicionado No 15 - Ao Vivo2482,375,575+58,46041,615,813', u'9=Mc Livinho - Fazer Falta311(x1)2,212,151-14,57363,656,312', u'10=Mc Gw - Ritmo Mexicano14102,190,060-17,80217,958,261', u'11=Simone e Simaria - Regime Fechado - Ao Vivo6292,134,182-36,38960,287,818', u'12+3Alok - Big Jet Plane1452,039,526+29,04124,353,302', u'13-1Henrique & Diego - Rasp\xe3o - Ao Vivo3861,989,428-132,47358,996,883', u'14+3Mc Kevinho - Encaixa2341,961,317+30,06440,816,203', u'15-2Camila Cabello - Havana223(x1)1,941,448-138,27632,732,070', u'16=Jorge & Mateus - Contrato9161,885,954-50,19813,645,614', u'17+1Henrique & Juliano - Vidinha de Balada - Ao Vivo4851,858,080+2,42174,575,207', u'18-41Kilo - Deixe Me Ir - Ac\xfastico331(x1)1,846,633-232,41054,211,335', u'19=MC Gustta - Abusadamente18161,801,312-36,79421,791,985', u'20+1Dua Lipa - New Rules262(x2)1,773,677-1,68140,925,866', u'21+6Gusttavo Lima - Apelido Carinhoso5211,730,116+131,6645,985,120', u'22-2Major Lazer - Sua Cara321(x9)1,723,947-55,02269,226,267', u'23+2Mc Livinho - Azul Piscina8231,686,818+25,8549,118,359', u'24-2IZA - Pesad\xe3o (Participa\xe7\xe3o especial Marcelo Falc\xe3o)13181,651,172-120,21716,223,284', u'25-2Pabllo Vittar - Corpo Sensual291(x2)1,637,805-57,89936,269,744', u'26-2Felipe Ara\xfajo - A Mala \xc9 Falsa - Ao Vivo54241,614,402-53,54331,901,280', u'27+2Felipe Ara\xfajo - Amor Da Sua Cama10271,608,747+34,8389,777,892', u'28-2Jo\xe3o Neto & Frederico - C\xea Acredita - Ao Vivo21171,608,168-5,21227,539,517', u'29+1MC G15 - Cara Bacana3071,552,613-9,03542,288,278', u'30+6Jerry Smith - Trof\xe9u do Ano9301,532,631+25,23810,460,880', u'31=Maiara & Maraisa - Sorte Que C\xea Beija Bem - Ao Vivo49111,510,392-49,67148,293,904', u'32+5Alok - Never Let Me Go3551,506,512+21,41353,436,147', u'33-5Cleber & Cauan - Quase - Ao Vivo16281,497,368-86,26015,420,230', u'34+1Wesley Safad\xe3o - Ressaca de Saudade - Ao Vivo24151,489,717-18,65029,397,372', u'35-1Post Malone - rockstar16141,486,153-25,04921,880,656', u'36+3Ed Sheeran - Shape of You521(x10)1,441,807+63,98799,112,623', u'37+1Pabllo Vittar - K.O.372(x3)1,359,303-47,35863,511,522', u'38-6MC Fioti - Bum Bum Tam Tam3201,347,117-200,2434,585,449', u'39+11Henrique & Diego - Escalada pra Jogar3391,329,193+235,9872,984,475', u'40=Mar\xedlia Mendon\xe7a - Transplante7401,318,605-53,5776,731,730', u'41=Gusttavo Lima - Eu Vou Te Buscar (Cha La La La La La)18331,305,458-45,73618,286,225', u'42=Selena Gomez - Wolves11141,261,027-3,47614,486,841', u'43+2Solange Almeida - Se \xe9 pra Gente Ficar19401,252,107+9,77816,346,111', u'44=Munhoz & Mariano - Mulher\xe3o da Porra8441,222,345-38,5156,991,623', u'45+2Alok - Hear Me Now631(x5)1,212,272+34,41994,895,931', u'46+2George Henrique & Rodrigo - De Copo Em Copo - Ao Vivo17461,206,433+34,59314,684,726', u'47-1Luis Fonsi - \xc9chame La Culpa7181,202,763-29,4739,364,112', u'48-15Mar\xedlia Mendon\xe7a - Amante N\xe3o Tem Lar - Ao Vivo51181,187,240-353,86842,120,495', u'49+26Gustavo Mioto - Coladinha em Mim - Ao Vivo2491,170,877+356,1251,985,629', u'50-7Kell smith - Era uma Vez27231,105,302-157,57025,543,955', u'51+6Mc Kevinho - Olha a Explos\xe3o533(x5)1,054,881+104,62156,672,207', u'52=Maroon 5 - What Lovers Do18211,043,039-11,22618,875,879', u'53-4Anavit\xf3ria - Fica - Bonus Track10241,026,107-109,90211,545,741', u'54-3Jorge & Mateus - Medida Certa33221,011,277-53,36933,283,662', u'55+5Jerry Smith - Pode Se Soltar3033983,651+76,84120,781,101', u'56-2Matheus & Kauan - Te Assumi Pro Brasil - Na Praia 2 / Ao Vivo606972,356-44,24172,164,093', u'57+5Mc Zaac - Vai Embrazando422(x2)971,903+70,10155,302,666', u'58+16Mc Kevinho - O Grave Bater414928,422+111,02548,849,636', u'59-4Henrique & Juliano - De Tr\xe1s Pra Frente - Ao Vivo2617923,719-56,40028,627,961', u'60-1Z\xe9 Felipe - Voc\xea N\xe3o Vale Nada - Ao Vivo1655920,302-15,48712,300,777', u'61+6Lucas Lucco - Tic Tac2217896,530+16,09023,122,759', u'62-1Matheus & Kauan - Nessas Horas - Ao Vivo3054879,950-25,51520,038,080', u'63+8Thiago Brava - Dona Maria1114878,674+17,2888,490,133', u'64=Anitta - Paradinha321(x6)871,288-22,07151,161,876', u'65-12Alesso - Is That For Me121(x1)870,296-179,92519,213,844', u'66-1Wesley Safad\xe3o - Ningu\xe9m \xc9 de Ferro4713860,997-28,88945,832,028', u'67-9Mar\xedlia Mendon\xe7a - De Quem \xc9 a Culpa? - Ao Vivo4322856,776-81,77433,706,682', u'68+2Luan Santana - Acordando o Pr\xe9dio558854,787-8,91154,990,136', u'69-6Bruno & Marrone - Na Conta Da Loucura - Ao Vivo1463851,999-44,2519,222,522', u'70-14Naiara Azevedo - Avisa Que Eu Cheguei - Ao Vivo2334850,113-103,85022,618,051', u'71-5Jorge & Mateus - Paredes - Ao Vivo9815847,261-36,68476,528,984', u'72+4J Balvin - Mi Gente277837,745+30,16631,349,574', u'73-5UM44K - 4 da manh\xe33133835,445-41,11622,106,475', u'74-5Nego do Borel - Esqueci Como Namora3015835,315-28,40432,982,784', u'75+2Mc Don Juan - Se Eu Tiver Solteiro1072809,599+11,5565,830,307', u'76-4Ivete Sangalo - Cheguei Pra Te Amar1041806,903-42,9518,932,092', u'77+7MC WM - Qual Bumbum Mais Bate1843789,758+43,95713,648,873', u'78=Jo\xe3o Neto & Frederico - Rapariga N\xe3o - Ao Vivo2278768,070-10,85411,043,532', u'79+4Mc Livinho - Esse Dom1839756,252+10,16814,533,260', u'80+6Mc L\xe9l\xe9to - Automaticamente4537735,547+31,59331,201,051', u'81+8Gustavo Mioto - Anti-Amor - Ao Vivo381732,234+36,6881,808,085', u'82-2Fernando & Sorocaba - Terapinga - Ao Vivo1780725,206-30,6899,858,084', u'83+2Felipe Ara\xfajo - Chave C\xf3pia - Ao Vivo4052716,760-23,90921,941,759', u'84-5Mar\xedlia Mendon\xe7a - Como Faz Com Ela - Ao Vivo8719710,616-55,67362,160,087', u'85-3Henrique & Juliano - Meu Amor - Ao Vivo2676701,617-45,96314,073,299', u'86+4MC WM - Sua Amiga Vou Pegar3528696,608+1,07528,925,534', u'87+20Alok - Villamix (Suave)2319690,498+41,53022,061,489', u'88+5Mc Don Juan - A Gente Brigou2880689,476+8,86814,763,870', u'89+3Marshmello - Silence2053688,785+2,40414,418,918', u'90+7Naiara Azevedo - Mentalmente - Ao Vivo2377684,632+10,77813,495,103', u'91+19The Chainsmokers - Something Just Like This4510683,268+43,64941,658,866', u'92+8Mc Don Juan - Lei do Retorno2942676,106+8,57818,598,548', u'93+9Hailee Steinfeld - Let Me Go1169675,813+17,3576,736,548', u'94+36Luis Fonsi - Despacito - Remix381(x6)674,274+161,80556,031,044', u'95-14Eminem - River357668,393-83,0202,302,776', u'96-9Henrique & Juliano - Aquela Pessoa - Ao Vivo2687665,785-37,79813,728,042', u'97+1Maluma - Felices los 43634665,295-5,93527,374,937', u'98+5Bonde R300 - Oh Nanana2958664,119+6,95218,178,778', u'99+9Nego do Borel - Voc\xea Partiu Meu Cora\xe7\xe3o511(x1)659,382+15,01560,949,344', u'100-12Hungria Hip Hop - Cora\xe7\xe3o de A\xe7o2467658,255-39,19615,503,862', u'101+16Alok - Fuego4421654,280+63,54230,354,092', u'102+3Naiara Azevedo - Pegada Que Desgrama2102652,743+1,7011,303,785', u'103-12Camila Cabello - Never Be the Same437649,769-41,0853,364,333', u'104-8Bruninho & Davi - E Essa Boca A\xed? - Ao Vivo3139649,226-26,90622,985,946', u'105+8Simone e Simaria - Loka524648,670+13,05256,690,900', u'106-12Ed Sheeran - Perfect2628646,014-31,83713,446,730', u'107+7Imagine Dragons - Thunder3629639,227+35,08426,948,411', u'108-9Sam Smith - Too Good At Goodbyes1730635,545-35,35612,929,158', u'109-8Demi Lovato - Sorry Not Sorry268633,209-29,47727,640,403', u'110+2ZAYN - Dusk Till Dawn1722619,683-17,99315,464,238', u'111+14Mc Denny - Vai Faz a Fila3111616,042+84,3711,578,047', u'112-8Luan Santana - Check-In458608,272-47,2512,838,185', u'113+7MC Fioti - Joga O Bum Bum Tam Tam3113604,606+17,6371,771,483', u'114-8Lil Pump - Gucci Gang1350598,333-51,6728,385,512', u'115+7Z\xe9 Neto & Cristiano - Amigo Taxista8115592,523+52,3843,606,989', u'116-21Mc Maha - Harry porra582588,015-89,1713,005,252', u"117-2Shawn Mendes - There's Nothing Holdin' Me Back3723577,742-16,07729,481,165", u'118-2Jonas Esticado - Com Amor N\xe3o Se Brinca - Ao Vivo9116577,273-13,7914,394,912', u'119-46Projota - Linda2526569,256-270,30723,196,986', u'120-11Jo\xe3o Neto & Frederico - Saudade do Caramba - Ao Vivo1668564,248-79,04810,286,814', u'121-2Henrique & Diego - Ciumento Eu - Ao Vivo5134563,689-24,44730,656,225', u'122-11Ed Sheeran - Perfect Duet (Ed Sheeran & Beyonc\xe9)511561,032-77,7814,847,284', u'123-5Jads & Jadson - Noites Frustradas - Ao Vivo9118558,924-29,8434,353,077', u'124+2Charlie Puth - Attention3638544,252+17,31825,610,093', u'125+2Wesley Safad\xe3o - Meu Cora\xe7\xe3o Deu PT - Ao Vivo727525,440+5,52551,275,289', u'126+2Calvin Harris - Feels2946525,053+5,98019,446,761', u'127-3DJ Khaled - Wild Thoughts2925524,166-11,39624,329,372', u'128-7Anavit\xf3ria - Trevo (Tu)5025521,904-32,28136,913,932', u'129+6Wesley Safad\xe3o - Sonhei Que Tava Me Casando - Ao Vivo19128518,328+28,5357,218,663', u'130-1Bruno Martini - Road1251515,967+3,4887,998,049', u'131-8Michel Tel\xf3 - Coisa de Deus - Ao Vivo1799508,560-28,4547,966,246', u'132+26The Chainsmokers - Closer751(x10)504,771+74,18663,507,908', u'133+21Danny Ocean - Me Reh\xfaso2594504,120+60,96411,937,523', u'134-3Post Malone - Congratulations36110503,758-3,87715,409,045', u'135-3NF - Let You Down1291492,223-9,7686,568,559', u'136-3CNCO - Reggaet\xf3n Lento (Remix)17120487,864-13,4727,637,068', u'137+14Axwell /\\ Ingrosso - More Than You Know2578476,857+27,26812,561,415', u'138-4Z\xe9 Neto & Cristiano - Cadeira de A\xe7o - Ao Vivo32134472,343-24,32211,198,383', u'139-3Marcos & Belutti - Solteiro Apaixonado42136471,192-16,57915,190,896', u'140+26Gabriel Diniz - Acabou Acabou (Quando Eu Digo Que Acabou) - Ao Vivo2140466,975+60,649873,301', u'141+9Luis Fonsi - Despacito493(x3)463,679+9,98742,821,161', u'142+1Imagine Dragons - Believer4839463,670-7,47727,085,475', u'143-5Fernando & Sorocaba - Bom Rapaz - Ao Vivo14138460,198-18,4555,813,880', u'144-3Anitta - Sim ou n\xe3o (Participa\xe7\xe3o especial Maluma)693(x1)457,142-15,88354,737,466', u'145-5Henrique & Diego - Ficadinha - Ao Vivo4783457,090-18,02718,770,457', u'146-9Naiara Azevedo - Mordida, Beijo e Tapa3286456,747-21,94914,095,730', u"147-2Bruno Mars - That's What I Like4621454,511-11,11628,397,108", u'148=Justin Bieber - Friends2012451,462-7,99515,154,344', u'149-7Matheus & Kauan - Exclusividade - Na Praia 2 / Ao Vivo4428449,132-22,16631,511,321', u'150-4Henrique & Juliano - Mais Amor e Menos Drama - Ao Vivo26124445,326-18,18210,833,010', u'151-12Jorge & Mateus - Pra Sempre Com Voc\xea - Ao Vivo9835445,307-31,24254,339,111', u'152-5Demi Lovato - Tell Me You Love Me1498445,280-16,3855,917,812', u'153+4Lil Uzi Vert - XO TOUR Llif33476443,126+7,51016,820,170', u'154+17Logic - 1-800-273-82552588441,807+41,39711,398,696', u'155-6Jorge & Mateus - Se o Amor Tiver Lugar4926438,336-16,77233,888,790', u'156+8Mano Walter - N\xe3o Deixo N\xe3o2156438,305+24,533852,077', u'157-5Maiara & Maraisa - Medo Bobo - Ao Vivo894434,637-11,60165,553,822', u'158-3BTS - MIC Drop6142433,039-6,4012,579,317', u'159-15Day e Lara - Enquadra9107432,856-38,2003,911,808', u'160+20J Balvin - Mi Gente1413428,219+43,49710,255,604', u'161+1Maluma - Coraz\xf3n4152423,618+4611,659,328', u'162-3MC WM - Aquecimento das Potrancas10142423,141-2,8134,039,493', u'163-71Kilo - Duro Igual Concreto2194419,563-17,9549,721,449', u'164-11Mar\xedlia Mendon\xe7a - Eu Sei de Cor - Ao Vivo | Ac\xfastico702(x5)419,554-24,05055,878,681', u'165-4Anitta - Movimento da Sanfoninha - Ao Vivo4155418,855-6,5241,498,318', u'166+13Mc Lan - Rabet\xe3o399418,113+32,36733,492,351', u'167+15G-Eazy - Him & I5145414,343+30,1471,951,457', u'168-5Turma do Pagode - Cobertor de Orelha - Ao Vivo2082411,447-11,24610,234,335', u'169-4Gaab - Tem Caf\xe97157407,616-9472,726,359', u'170+5Bruno Martini - Sun Goes Down2863401,440+13,23615,254,328', u'171-11Pineapple StormTv - Poesia Ac\xfastica #3: Capricorniana7138400,478-25,4433,002,384', u'172=Henrique & Diego - Malbec (Part. Dennis Dj) - Ao Vivo3817395,238-4,22027,736,759', u'173+25Atitude 67 - Cerveja De Garrafa (Fuma\xe7a Que Eu Fa\xe7o) - Ao Vivo2173393,566+37,997749,135', u'174-1David Guetta - 2U306393,316-5,58121,951,041', u'175-6Matheus & Kauan - Decide A\xed - Na Praia / Ao Vivo918391,839-10,92860,041,782', u'176-9Ed Sheeran - Photograph15311388,460-17,78252,956,362', u'177-9Shawn Mendes - Treat You Better8314384,717-18,94643,712,244', u'178+15Bebe Rexha - Meant to Be9136380,086+16,8833,641,968', u'179-2Maiara & Maraisa - 10% - Ao Vivo874379,566-8,01468,787,203', u'180REDennis DJ - Um Brinde4180379,1321,371,932', u'181REDimmi - Found U2181375,266700,332', u'182-12Taylor Swift - Look What You Made Me Do191(x1)374,774-25,86419,778,812', u'183-7Maroon 5 - Cold9154374,468-13,1863,389,404', u'184REMc Kevinho - T\xf4 Apaixonado Nessa Mina2516372,45419,704,541', u'185REPortugal. The Man - Feel It Still4182371,3021,334,585', u'186-3Shakira - Perro Fiel12129370,501-11,1894,824,715', u'187-3Avicii - Without You2188370,048-1,2289,518,516', u'188-10Projota - Oh Meu Deus4964366,977-19,40425,819,133', u"189REThe Chainsmokers - Don't Let Me Down8810362,94747,734,793", u'190=Nego do Borel - Contatinho3166362,028-3,5281,141,886', u'191+3Mc Gw - Atura Ou Surta 28179359,852-5712,777,706', u'192-4The Weeknd - Starboy672(x1)359,554-7,77349,523,156', u'193-1Perera DJ - Os Maloca1397359,392-4,7396,556,887', u'194-13Selena Gomez - Fetish2610359,283-25,05416,770,093', u'195-8Claudia Leitte - Baldin de Gelo2280357,872-11,64111,621,602', u'196=Avicii - Lonely Together13130357,140-4295,281,148', u'197RELudmilla - Cheguei3121353,28623,257,817', u'198-24Hungria Hip Hop - N\xe3o Troco691352,881-40,2472,810,291', u'199NEWJerry Smith - Nossa Que Absurdo1199352,286352,286', u'200-1MC Gudan - Boca de P\xealo9144350,693-2,7423,455,285']

SIM · Answer 4 · 2018-01-08T09:52:40.517

This can be another way to get data from the first column of that table:

import requests
from bs4 import BeautifulSoup

page1 = requests.get("http://kworb.net/spotify/country/br_weekly.html")                                                  
soup = BeautifulSoup(page1.text, 'lxml')
for tr in soup.select("tbody tr"):
    tds = ' '.join([item.text for item in tr.select('td')[:1]])
    print(tds)

Partial output:

html table scraping in python

4 Answers4