Regexp for finding href in open-uri ruby

Question

I need to find distance between two websites useing ruby open-uri. Using

def check(url)
    site = open(url.base_url)
    link = %r{^<([a])([^"]+)*([^>]+)*(?:>(.*)<\/\1>|\s+\/>)$}
    site.each_line {|line| puts $&,$1,$2,$3,$4 if (line=~link)}
    p url.links
end

Finding links not working properly. Any ideas why ?

Answer 1

If you want to find the a tags' href parameters, use the right tool, which isn't often a regex. More likely you should use a HTML/XML parser.

Nokogiri is the parser of choice with Ruby:

require 'nokogiri'
require 'open-uri'

doc = Nokogiri.HTML(open('http://www.example.org/index.html'))
doc.search('a').map{ |a| a['href'] }

pp doc.search('a').map{ |a| a['href'] }
# => [
# =>  "/",
# =>  "/domains/",
# =>  "/numbers/",
# =>  "/protocols/",
# =>  "/about/",
# =>  "/go/rfc2606",
# =>  "/about/",
# =>  "/about/presentations/",
# =>  "/about/performance/",
# =>  "/reports/",
# =>  "/domains/",
# =>  "/domains/root/",
# =>  "/domains/int/",
# =>  "/domains/arpa/",
# =>  "/domains/idn-tables/",
# =>  "/protocols/",
# =>  "/numbers/",
# =>  "/abuse/",
# =>  "http://www.icann.org/",
# =>  "mailto:iana@iana.org?subject=General%20website%20feedback"
# => ]

Answer 2

I see several issues with this regular expression:

• It is not necessarily the case that a space must come before the trailing slash in an empty tag, yet your regexp requires it

• Your regexp is very verbose and redundant

Try the following instead, it will extract you the URL out of <a> tags:

link = /<a \s   # Start of tag
    [^>]*       # Some whitespace, other attributes, ...
    href="      # Start of URL
    ([^"]*)     # The URL, everything up to the closing quote
    "           # The closing quotes
    /x          # We stop here, as regular expressions wouldn't be able to
                # correctly match nested tags anyway