-1

I am trying to combine a batch search for some DNS records search with Whois search on my terminal. I have a CSV file with a list of domains and I would like to run the following batch searchers:

  • MX search: host -t mx $domain
  • NS search: host -t ns $domain

This are pretty easy.

Combine this with Whois Search ; Which returns only a summary of some of the Whois data;I would need to query the whois server for the domain which is fine like: whois

I can use -h, to only record Domain Registrant Details such as Telephone, Country code etc. I have tried this:

  • Whois: whois -h 'Registrar WHOIS Server:' "domain" Which gives me the output as well for only Registrant details.

So when I combine all into a single bash file, I get:

#!/usr/bin/env bash

file="${1:-input_test1.csv}"

if [[ ! -f "$file" ]]; then
    printf 'No file: %s\n' "$file" >&2
    exit 1
fi

(
  read -r header; printf '%s\n' "$header"
  while IFS=, read -r domain; do
    mx="$(host -t mx "$domain" | sort | head -1)"
    ns="$(host -t ns "$domain" | sort| head -1)"
    whois="$(whois -h "$(whois" "$domain" | grep 'Registrar WHOIS Server:') "$domain")
    printf '%s,"%s"\n' "$domain" "$mx" "$ns" "$whois"
  done
) < "$file"

I would love to get a CSV output with the domain, mx (only 1), NS (only 1), whois whois is registrant data s shown below;

Sample Expected Output Screengrab

Thank you.

oslee
  • 11
  • 7

1 Answers1

0

You already know that different domains point to different whois servers. I think you are going to find that each registrar has their own favourite way of presenting information via whois, and that they are not consistent. ICANN mandates that a minimum set of data be available via whois, but some of the data you're looking for may fall outside that set.

The following strips just basic data from whois.internic.net, which you can use for gathering DNS servers, whois servers and MX:

#!/usr/bin/env bash

mapfile -t domains < domains.lst

declare -i i
for this in "${domains[@]}"; do
  unset a; declare -A a=()
  unset ns; declare -a ns=()
  whois=""
  i=0
  while IFS=: read -r key value; do
    #printf "key=%s / value=%s\n" "$key" "$value"
    case "$key" in
      *"Registrar WHOIS Server") whois="${value## }" ;;
      *"Name Server") ns+=("${value## }") ;;
    esac
  done < <(whois -h whois.internic.net "$this")
  read mx < <(host -t mx "$this" | sort | awk 'NR==1{print $NF}')

  printf '%s,%s,%s,%s\n' \
    "$this" \
    "$mx" \
    "$whois" \
    "$(printf '%s ' "${ns[@]}")"
done

If you really want to try to scrape from the whois data at $whois, the script above should show you how you might be able to do that for each domain in your list.

ghoti
  • 45,319
  • 8
  • 65
  • 104
  • Hi Ghoti, thank you very much for your elegant solution. I have been trying to run it with a test file named test1.lst and I keep getting an error. `hunt2.sh: line 2: mapfile: command not found hunt2.sh: line 16: syntax error near unexpected token `<' hunt2.sh: line 16: ` done < <(whois -h whois.internic.net "$this")` – oslee Jan 21 '19 at 06:56
  • Ah, it's possible you're using an older version of bash. The `mapfile` command was introduced with bash 4, which is about a decade old. As far as I'm aware, the only modern operating system still shipping bash 3 is macOS. If you're using macOS, you could install bash 4 using [brew](http://brew.sh) or [MacPorts](http://macports.org). For future reference, it's usually a good idea to put the version of what you're using in your question, to avoid getting stumped by assumptions. – ghoti Jan 21 '19 at 07:01
  • Thank you ghoti, you are far too kind. I was abke to get the script up and running once i updated by bash to 5. – oslee Jan 21 '19 at 14:44
  • The script is running and printing the output on the terminal but not writing into the file. Is there something I need to do to pipe the output to the file or to a new one? managed to get luck by running a single liner and getting much more DNS data with `whois -h $(whois example.com | grep 'WHOIS Server:' | cut -f2- -d:) example.com' and i was hoping to modify the section here with that at: done `< <(whois -h whois "$this" "$this")` taking a look at [rfc1036/whois](https://github.com/rfc1036/whois/commit/89d2aef1581df9038de4d9e81fab239b03f9ea8c) – oslee Jan 21 '19 at 15:05
  • My pleasure - it's my hope that everyone have a long and useful relationship with StackOverflow, and high-quality long-lived questions are a vital part of that. :) That said, it's very difficult to parse code that is included in comments. If you think this code makes sense as part of THIS question, please edit the question and add your clarification. If you think it's a different question, do [ask a new question](https://stackoverflow.com/questions/ask). A single question with too many parts is unlikely to get a complete answer. – ghoti Jan 21 '19 at 17:18