Parse HTML in Enlive like in Beautifulsoup

Question

I am trying to get links from HTML in Clojure with Enlive. Can I get a list of all links from a page? Can I iterate over them?

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc, 'html.parser')

print(soup.prettify())
# <html>
#  <head>
#   <title>
#    The Dormouse's story
#   </title>
#  </head>
#  <body>
#   <p class="title">
#    <b>
#     The Dormouse's story
#    </b>
#   </p>
#   <p class="story">
#    Once upon a time there were three little sisters; and their names were
#    <a class="sister" href="http://example.com/elsie" id="link1">
#     Elsie
#    </a>
#    ,
#    <a class="sister" href="http://example.com/lacie" id="link2">
#     Lacie
#    </a>
#    and
#    <a class="sister" href="http://example.com/tillie" id="link2">
#     Tillie
#    </a>
#    ; and they lived at the bottom of a well.
#   </p>
#   <p class="story">
#    ...
#   </p>
#  </body>
# </html>

links = soup.find_all('a')

or

links = soup('a')

How can I do this in Clojure with Enlive?

look at `enlive/select` and `[:a]` https://github.com/cgrand/enlive#selector-syntax — nha, Feb 05 '18 at 12:24

score 2 · Answer 1 · answered Feb 05 '18 at 13:13

First you need to ingest some HTML using Enlive's html-resource function. We'll grab news.google.com:

(defn fetch-url [url]
  (html/html-resource (java.net.URL. url)))
(def goog-news (fetch-url "https://news.google.com"))

To get all the <a> tags, use select function with a simple selector (the second argument):

(html/select goog-news [:a])

This will evaluate to a sequence of maps, one per <a> tag. Here's an example <a> tag map from today's news:

{:tag :a,
 :attrs {:class "nuEeue hzdq5d ME7ew",
         :target "_blank",
         :href "https://www.vanityfair.com/hollywood/2018/01/first-black-panther-reviews",
         :jsname "NV4Anc"},
 :content ("The First Black Panther Reviews Are Here—and They're Ecstatic")}

To get the inner text of each <a>, you can map Enlive's text function over the results e.g. (map html/text *1). To get each href, you can (map (comp :href :attrs) *1).

score 1 · Accepted Answer · edited May 09 '19 at 19:01

that would be very simple:

(require '[net.cgrand.enlive-html :as enlive])

(let [data (enlive/html-resource (java.net.URL. "https://www.stackoverflow.com"))
      all-refs (enlive/select data [:a])]
  (first all-refs))

;;=> {:tag :a, :attrs {:href "https://stackoverflow.com", :class "-logo js-gps-track", :data-gps-track "top_nav.click({is_current:true, location:1, destination:8})"}, :content ("\n                   " {:tag :span, :attrs {:class "-img"}, :content ("Stack Overflow")} "\n                ")}

the all-refs collection would contain all the links from page in enlive representation form.

(let [data (enlive/html-resource (java.net.URL. "https://www.stackoverflow.com"))
      all-refs (enlive/select data [:a])]
  (map #(-> % :attrs :href) all-refs))

would for example collect all the href values from links

Parse HTML in Enlive like in Beautifulsoup

2 Answers2