0

I am trying to parse just the index page of every domain on the web. The only alternative that I can think of this that would produce somewhat acceptable results would be a general web crawl to identify every unique domain on the web.

Instead, I would like to be able to just iterate through a list of all domain names on the web. Does a public (or private but not incredibly expensive) database dump like this exist?

Related question that may answer this question: is it possible to host your own DNS server without querying any other dns servers?

I suppose that looping through every possible ipv4 combination is also an option but that would be less targeted than the set of all registered domains.

Avi Mosseri
  • 1,258
  • 1
  • 18
  • 35
  • "is it possible to host your own DNS server without querying any other dns servers?" Yes, you can install any authoritative nameserver you want locally. It is also mostly useless (specially outside of the LAN) since noone will query it. But based on how you phrase the question I suspect you may not already know the DNS and the difference between an authoritative and recursive nameserver. So for more help you will need to rephrase your question, and post it on another forum like [su] or [sf] (read their help) as this is offtopic here. – Patrick Mevzek Feb 08 '19 at 15:57
  • "I suppose that looping through every possible ipv4 combination" 1) You are aware of IPv6 ? 2) You are aware of multihoming ? Doing an HTTP connection to a port 80 or even 443 port is not enough, what will you put inside the `Host` header (or the TLS SNI extension)? – Patrick Mevzek Feb 08 '19 at 15:57
  • All DNS records in existence is NOT a list of all domain->ip pairs. The DNS does not store just that mapping and have others records beside `A` or `AAAA`. Also it is not (domain,IP) pairs as they could be multiple IPs (even in one given family) so in the DNS world we speak about set of records. – Patrick Mevzek Feb 08 '19 at 16:08

1 Answers1

4

I think I may have found a solution: czds.icann.org/

Avi Mosseri
  • 1,258
  • 1
  • 18
  • 35
  • This gives you data only about gTLDs, nothing about ccTLDs. – Patrick Mevzek Feb 08 '19 at 15:53
  • I submitted a separate application for .com domains that should be processed within 2 weeks. I will also be tediously filling out applications for all other major domains. – Avi Mosseri Feb 08 '19 at 22:51
  • Make sure to read with attention the terms of service you sign. You are not allowed to do everything with this list. Also it all depends on your definition of "major". If it is by pure volume of domains, `.CN` has more than 20 millions of domains which is far more than many gTLDs (in fact it is second by volume, just after `.COM`). `.DE` is third with 16 millions, etc. – Patrick Mevzek Feb 08 '19 at 23:05