How to search through links and save only those that contain a specific data?

Question

I have a text file with thousands of hyperlinks in the format "URL = http://examplelink.com" in a file called mylinks.txt.

What I want to do is search through all of these links, and checks if any of them contains some keywords, like, "2018", "2017". If the link contains the keyword, I want to save the link in the file "yes.txt" and if it doesn't it goes to the file "no.txt".

So at the end, I would end up with two files: one with the links that send me to pages with the keywords I'm searching for, and other one with the links that doesn't.

I was thinking about doing this with curl, but I don't know even if it's possible and I don't know also how to "filter" the links by keywords.

What I have got until now is:

curl -K mylinks.txt >> output.txt

But this only creates a super large file with the HTML's of the links it searches. I've searched and read through various curl tutorials and haven't found anything that "selectively" search for pages and save the links (not the content) of the pages it found matching the criteria.

Answer 1

-Untested- For links in lines containing "2017" or "2018".

cat mylinks.txt | grep -E '2017|2018' | grep -o 'URL =*>' >> yes.txt

To get url of lines that doesn't contain the keywords.

cat mylinks.txt | grep -vE '2017|2018' | grep -o 'URL =*>' >> no.txt

This is unix piping. (The char | ) takes the program output stdout at the left and feed the stdin to the program on the right.

In Unix-like computer operating systems, a pipeline is a sequence of processes chained together by their standard streams, so that the output of each process (stdout) feeds directly as input (stdin) to the next one. https://en.wikipedia.org/wiki/Pipeline_(Unix)

Answer 2

Here is my take at it (kind of tested on an url-file with a few examples). This is supposed to be saved as a script, it's too long to type it into the console directly.

#!/bin/bash
urlFile="/path/to/myLinks.txt"
cut -d' ' -f3 "$urlFile" | \
while read url
do
  echo "checking url $url"
  if (curl "$url" | grep "2017") 
  then 
    echo "$url" >> /tmp/yes.txt
  else 
    echo "$url" >> /tmp/no.txt
  fi
done

Explanation: the cut is necessary to cut away the prefix "URL = " in each line. Then the url's are fed into the while-read loop. For each url, we curl it, grep for the interesting keyword in it (in this case "2017"), and if the grep returns 0, we append this URL to the file with the interesting URLs.

Obviously, you should adjust the paths and the keyword.