Since I didn't understand which of these behaviors you are asking for, I post solutions to different cases here:
Get links, order by highest vote:
SELECT votes.link_id, links.url, MAX(votes.vote) AS maxvote
FROM votes INNER JOIN links ON links.id = votes.link_id
GROUP BY votes.link_id ORDER BY `maxvote` DESC;
Get links, order by average vote:
SELECT votes.link_id, links.url, AVG(votes.vote) AS avgvote
FROM votes INNER JOIN links ON links.id = votes.link_id
GROUP BY votes.link_id ORDER BY `avgvote` DESC;
Get links, order by votes count:
SELECT votes.link_id, links.url, COUNT(votes.vote) AS cntvotes
FROM votes INNER JOIN links ON links.id = votes.link_id
GROUP BY votes.link_id ORDER BY `cntvotes` DESC;
Get votes, by highest vote:
SELECT links.id, links.url, votes.vote
FROM links INNER JOIN votes ON links.id = votes.link_id
ORDER BY votes.vote DESC;
(I omitted some fields to shorten queries, of course feel free to add them back again).