How do I get the absolute path for '' in node from the a response.body

Question

So I want to use request-promise to pull the body of a page. Once I have the page I want to collect all the tags and get an array of src's of those images. Assume the src attributes on a page have both relative and absolute paths. I want an array of absolute paths for imgs on a page. I know I can use some string manipulation and the npm path to build the absolute path but I wanted to find a better way of doing it.

var rp = require('request-promise'),
    cheerio = require('cheerio');

var options = {
    uri: 'http://www.google.com',
    method: 'GET',
    resolveWithFullResponse: true
};

rp(options)
  .then (function (response) {
    $ = cheerio.load(response.body);
    var relativeLinks = $("img");
    relativeLinks.each( function() {
        var link = $(this).attr('src');
        console.log(link);
        if (link.startsWith('http')){
            console.log('abs');
        }
        else {
            console.log('rel');
        }
   });
});

results

  /logos/doodles/2016/phoebe-snetsingers-85th-birthday-5179281716019200-hp.gif
  rel

Possible duplicate of [getting the absolute path of a ](http://stackoverflow.com/questions/3496491/getting-the-absolute-path-of-a-img) — Midas, Jun 09 '16 at 19:01
@Midas This question is closely related, but not quite a duplicate of that other question because of the implementation differences between the DOM and jQuery in that case, and Cheerio in this case. Doing something like $(this) or $('img')[0].src won't return anything in Cheerio. — Michael, Jun 10 '16 at 17:02

score 4 · Answer 1 · answered Jun 09 '16 at 19:14

Store your page URL as a variable use url.resolve to join the pieces together. In the Node REPL this works for both relative and absolute paths (hence the "resolving"):

$:~/Projects/test$ node
> var base = "https://www.google.com";
undefined
> var imageSrc = "/logos/doodles/2016/phoebe-snetsingers-85th-birthday-5179281716019200-hp.gif";
undefined
> var url = require('url');
undefined
> url.resolve(base, imageSrc);
'https://www.google.com/logos/doodles/2016/phoebe-snetsingers-85th-birthday-5179281716019200-hp.gif'
> imageSrc = base + imageSrc;
'https://www.google.com/logos/doodles/2016/phoebe-snetsingers-85th-birthday-5179281716019200-hp.gif'
> url.resolve(base, imageSrc);
'https://www.google.com/logos/doodles/2016/phoebe-snetsingers-85th-birthday-5179281716019200-hp.gif'

Your code would change to something like:

var rp = require('request-promise'),
    cheerio = require('cheerio'),
    url = require('url'),
    base = 'http://www.google.com';

var options = {
    uri: base,
    method: 'GET',
    resolveWithFullResponse: true
};

rp(options)
  .then (function (response) {
    $ = cheerio.load(response.body);
    var relativeLinks = $("img");
    relativeLinks.each( function() {
        var link = $(this).attr('src');
        var fullImagePath = url.resolve(base, link); // should be absolute 
        console.log(link);
        if (link.startsWith('http')){
            console.log('abs');
        }
        else {
            console.log('rel');
        }
   });
});

Nick Bartlett · Accepted Answer · 2016-06-09T19:56:03.250

To get an array of image links in your scenario, you can use url.resolve to resolve relative src attributes of img tags with the request URL, resulting in an absolute URL. The array is passed to the final then; you can do other things with the array other than console.log if so desired.

var rp = require('request-promise'),
    cheerio = require('cheerio'),
    url = require('url'),
    base = 'http://www.google.com';

var options = {
    uri: base,
    method: 'GET',
    resolveWithFullResponse: true
};

rp(options)
    .then (function (response) {
        var $ = cheerio.load(response.body);

        return $('img').map(function () {
            return url.resolve(base, $(this).attr('src'));
        }).toArray();
    })
    .then(console.log);

This url.resolve will work for absolute or relative URLs (it resolves and returns the combined absolute URL when resolving from your request URL to a relative path, but when resolving from your request URL to an absolute URL it just returns the absolute URL). For example, with img tags on google with /logos/cat.gif and https://test.com/dog.gif as the src attributes, this would output:

[ 
    'http://www.google.com/logos/cat.gif',
    'https://test.com/dog.gif'
]

score 0 · Answer 3 · edited Jun 09 '16 at 18:59

0

It looks like you're using jQuery, so you could

$('img').each(function(i, e) {
    console.log(e.src)
});

If you use src it will expand relative paths to absolute ones.

edited Jun 09 '16 at 18:59

Midas

7,012
5
34
52

answered Jun 09 '16 at 18:51

skalpin

310
1
10

Correct, but `src` is not a jQuery attribute. It's standard Javascript. – Midas Jun 09 '16 at 18:55
2

It's a DOM attribute - you don't have to use jQuery for _everything_ – Ryan Wheale Jun 09 '16 at 18:56
To be a little pedantic, he's using Cheerio, which is a NodeJS module that uses jQuery-like syntax for traversing the DOM. Also, in this case accessing the src attribute won't help because it will still return a root-relative URL (or whatever is in the DOM) when he needs the absolute path. – Michael Jun 09 '16 at 19:06

Oleksandr.Bezhan · Answer 4 · 2022-10-13T11:31:43.653

It's 2022 and url.resolve is deprecated now.

Here is how I do it (works both for a 'href' and img 'src'):

import URI from 'urijs'

function absolutizeUri(maybeRelativeUri: string, baseUri: string): string {
    if (!maybeRelativeUri || maybeRelativeUri.length === 0) {
        return ''
    }
    let uri = new URI(maybeRelativeUri);
    if (uri.is('relative')) {
        uri = ur.absoluteTo(baseUri)
    }
    return uri.toString()
}

// ...
const baseUri = 'http://www.google.com'
const src = absolutizeUri($(this).attr('src'), baseUri)

How do I get the absolute path for '' in node from the a response.body

4 Answers4

Linked