2

I'm trying to access hyperlink texts that will always be stored as the last hyperlink tag nested in 100 div tags on a certain website. In the below example, "00A17" would be what I'm trying to extract from a single tag.

HTML Code: 

<div class="headlineText">
  <a class="mrnum" title="Full MathSciNet Item" href="/mathscinet/search/publdoc.html?arg3=&amp;co4=AND&amp;co5=AND&amp;co6=AND&amp;co7=AND&amp;dr=all&amp;extend=1&amp;l=100&amp;pg4=AUCN&amp;pg5=TI&amp;pg6=PC&amp;pg7=ALLF&amp;pg8=ET&amp;review_format=html&amp;s4=&amp;s5=&amp;s6=&amp;s7=%22Featured%20Review%22&amp;s8=Journals&amp;sort=Newest&amp;vfpref=html&amp;yearRangeFirst=&amp;yearRangeSecond=&amp;yrop=eq&amp;r=9&amp;mx-pid=2650657"><strong>MR2650657</strong></a>
  <a class="item_status" href="/mathscinet/help/fullitem_help_full.html#indexed"><span>Indexed</span></a>
  <a href="/mathscinet/search/author.html?mrauthid=115370">Logan, J. David</a> <span class="title"><span class="searchHighlight">Featured review</span>: <span class="it">Introduction to the foundations of applied mathematics</span> [<a href="/mathscinet/search/publdoc.html?pg1=MR&amp;s1=2526777&amp;loc=fromrevtext">MR2526777</a>].</span>
  <a href="/mathscinet/search/journaldoc.html?id=5174"> <em>SIAM Rev.</em></a> <a href="/mathscinet/search/publications.html?pg1=ISSI&amp;s1=281478"> 52 </a>
  <a href="/mathscinet/search/publications.html?pg1=ISSI&amp;s1=281478"> (2010), </a>
  <a href="/mathscinet/search/publications.html?pg1=ISSI&amp;s1=281478"> no. 1,</a> 173–178.
  <a href="/mathscinet/search/mscdoc.html?code=00A17">00A17</a>
</div>

The code I wrote to attempt this is basically a mess

Python Code for remote access:

headlineTexts = []
mscLinks = []
for x in range(2,102):
    headlineTexts.extend(driver.find_elements_by_xpath("//*[@id='content']/form/div[3]/div[2]/div/div/div[%d]/div[2]"%x))
    mscLinks.extend(headlineTexts[x-2].find_elements_by_tag_name("a")[last()])

Essentially, there are 100 "headlineText" div tags (indexed from 2 to 101), and I need to get the aforementioned hyperlink text out of each one (hence the iteration). I'm basically creating a list of headlineText elements and then for each headlineText element I'm attempting to extract the last subelement with the tag name "a". Unfortunately, when I attempt to run this I get a

TypeError: 'WebElement' object is not iterable

which is strange to me because I'm only using the plural find_elements() and that should generate an iterable list. Am I perhaps using last() incorrectly?

Aaron Cao
  • 69
  • 2
  • 7
  • try to get all the href by using the anchor tag.`headlineTexts.extend(driver.find_elements_by_tagname("a")) – Alok Jan 18 '18 at 12:18

1 Answers1

4

To retrieve the hyperlink text that will always stored as the last hyperlink tag nested in 100 div tags on a certain website e.g. 00A17 you can use the following line of code :

print(driver.find_element_by_xpath("div[@class='headlineText']//a[last()]").get_attribute("innerHTML"))
undetected Selenium
  • 183,867
  • 41
  • 278
  • 352