detect external broken links

Question

The problem is that I have on my website many of exetrnal download links, and some of the links get expired, so I want to detect automatically the expired links.

for me a valid link is a direct file download link pointing to one of my file servers. a broken link lead to a simple html page with an error message.

my first idea was to get the html source code of the download link and see if it contains an error but it did not work. I've tried also javascript but the problem is that js do not deal with external links.

any ideas?? thanks a lot

You could always send an AJAX request for every URL you're looking at, and see if you get a 404, 500, etc. error. — Ian, Sep 30 '12 at 16:01
doing this clientside for every link and for every client which visits your site is a big overhead, why not check it once a day on the serverside if a link is broken and remove it from the database? — supernova, Sep 30 '12 at 16:02

Alcides Queiroz · Answer 1 · 2012-10-03T12:16:38.380

2

This isn't a task for your front-end, but for the back-end. As supernova said, check it from your server once a day. AJAX requests will not be your answer, since the browser security policy doesn't allow requests to different domains.

Solution:

Ok, based on your comment, check this solution:

<html>
<head>
  <script src='http://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js'></script>
  <script>
    $(document).ready(function(){
      var linksDiv = $('#links');

      $('#generateLinks').click(function(){
        //I don't know your logic for this function, so I'll try to reproduce the same behavior
        var someURLs = ['http://www.google.com','http://www.djfhdkjshjkfjhk.com', 'http://www.yahoo.com'];
        linksDiv.html('');

        for(var i = 0; i < someURLs.length; i++){
          var link = $('<a/>').attr('href', someURLs[i]).append('link ' + i).css('display','block');
          linksDiv.append(link);
        }

      });

      $('#getLinksAndSend').click(function(){
        var links = linksDiv.find('a');

        var gatheredLinks = [];

        $(links).each(function(){
          gatheredLinks.push(this.href);
        });

        sendLinks(gatheredLinks);
      });

      var sendLinks = function(links){
        $.ajax({
          url: "your_url",
          type: "POST",
          data: {
            links: links
          }
        }).done(function(resp) { 
          alert('Ok!')
        });
      }
    });
  </script>
</head>
<body>
  <div id="links">

  </div> 
  <button id="generateLinks">Generate all links</button>
  <button id="getLinksAndSend">Get links and send to validator</button>
</body>
</html>

edited Oct 03 '12 at 12:16

answered Sep 30 '12 at 16:07

Alcides Queiroz

9,456
3
28
43

My aim is to have a button that once clicked it scan all the links I have on given page and show the broken links, thats all. – user1709976 Sep 30 '12 at 16:22
Ok, your scenario is exactly the same I mentioned above, you'll need to do this check in your server, since cross-domain requests are not possible through AJAX. If the links you want to check are under the same domain your app is hosted, you'll be able to do it just using JS, but I think this is not your case. What language do you use in your server? – Alcides Queiroz Sep 30 '12 at 16:35
I think this is what you need: http://www.designaeon.com/check-dead-links-from-database-using-php-curl/ – Alcides Queiroz Sep 30 '12 at 16:45
Im having trubles with cURL if you have any other suggestions?? – user1709976 Oct 01 '12 at 12:09
Thanks for the reply, I've adapted your script but I dont know why it did not redrict to the page to wich I want to send data via POST. – user1709976 Oct 03 '12 at 12:25
My solution is using AJAX, so it won't redirect to any URL, but it will send your data silently and receive a response from the server. – Alcides Queiroz Oct 03 '12 at 12:43
Sorry for my newby question, but Id like to recieve back explicitly the response, how can I do that ? – user1709976 Oct 03 '12 at 13:02
This way you'll receive your response, but through the callback passed to "done". In my example I put a simple alert there, but you can do what you want in that function. The resp parameter is your server response, which can be any text (JSON, XML or a normal text). Do you have some doubt about how to work with asynchronous requests? – Alcides Queiroz Oct 03 '12 at 14:56
yes I'm not too familiar with this technology, Im struggling to show the result of the php script back...? – user1709976 Oct 04 '12 at 14:39

score 2 · Answer 2 · edited May 23 '17 at 11:46

if you dont mind letting the client do the work, you could try doing it with javascript.

i have a greasemonkey script that automatically checks all links in the open page, and mark them according to the server response (not found, forbidden, etc).

see if you can get some ideas from it: http://userscripts.org/scripts/show/77701

i know that cross domain policies do not apply to GM_xmlhttprequest, and if want to use a javascript solution, might have to try a workaround, like:

if you want a server side solution, i believe the above answer can help you.

score 0 · Answer 3 · answered Apr 20 '17 at 18:12

0

It may be overkill but there's a program in linux kde called klinkstatus that can find broken links in a website:

https://www.kde.org/applications/development/klinkstatus/

answered Apr 20 '17 at 18:12

user3825550

11
3

detect external broken links

3 Answers3