2

Using the HttpWebRequest (even when calling the async get method), it can take up to 15 seconds due to DNS resolution:

A Domain Name System (DNS) query may take up to 15 seconds to return or time out.

Is it possible to make this query in advance, so that it wont spend time at it when making the request?

Jonathon Reinhart
  • 132,704
  • 33
  • 254
  • 328
ebb
  • 9,297
  • 18
  • 72
  • 123
  • 1
    It DNS resolution is taking 15 seconds, *something is wrong*, and it will probably be wrong if you try to do it ahead of time as well. – Jonathon Reinhart May 13 '14 at 06:43
  • @JonathonReinhart, `up to` - It is not necessarily taking 15 seconds. Given a large set of urls to request, it would be a lot faster if the DNS queries could be made in advance. – ebb May 13 '14 at 06:49
  • In advance of... the web requests? That happens anyway. If you add up all the time you spend processing, it's still accounted for. It only makes sense to pre-fetch something if you can do it before you need it. – Jonathon Reinhart May 13 '14 at 06:51
  • 1
    @JonathonReinhart, In advance of the actual web request, yes. - I want to make the DNS queries on another server, fetch them on a second which then will make the web requests. This way I'm able to "distribute" the work. – ebb May 13 '14 at 06:57

1 Answers1

1

It is very likely that the DNS resolution in HttpWebRequest is not pluggable. But you can do the resolution yourself and rewrite the URLs to use the resolved IP as the host name. Make sure to set the correct host-header, though, because many websites require it. Also, I believe HTTPS will be harder because certificate validation is based in part on the host name.

usr
  • 168,620
  • 35
  • 240
  • 369
  • That's a pretty neat idea. Do I have to use the IP also after the first request, in order to "skip" the dns resolution - or short: Do I need to use the IP to navigate the website? – ebb May 13 '14 at 11:14
  • If you specify a host name HttpWebRequest must resolve it to an IP by itself. Don't you think? – usr May 13 '14 at 11:20
  • Generally yes, however I was thinking if there was any kind of cache I could tamper with, so that `HttpWebRequest` didn't have to resolve it to an IP. – ebb May 13 '14 at 11:22
  • I see. Well, you can take a look using Reflector or the reference source. Probably, all Microsoft code uses the machines DNS resolver and cache so that it cooperates responsibly with other code on the system. I have two more ideas: Add resolved names to the hosts file, use a custom DNS server (probably not as hard as it sounds, see https://www.google.com/webhp?complete=1&hl=en#complete=1&hl=en&q=c%23+dns+server). – usr May 13 '14 at 11:24