0

I am running hundreds of parallel HTTP requests for all different domains using node http module. If I understood it correctly, node http module does dns lookup using worker threads and hence number of lookups at a time are limited by libuv threads. In order to speed up execution and avoid this supposed congestion, i supplied my custom lookup function which used dns resolve.

However, I am surprised that total time to process (receive http full response i.e res end event) 300 http requests increased from 1.4 seconds (when using default lookup function) to 6 seconds (when using dns resolve)

My custom look up function

const staticLookup = () => async (hostname: string, _: null, cb: Function) => {
                console.log("DNS look up started for: " + hostname);
                // const ips = await dnsPromises.resolve(hostname);
                dns.resolve(hostname, (err, records) => {
                    const ips = records;
                    // console.log("ips: " + JSON.stringify(ips));
                    if (ips.length === 0) {
                        throw new Error(`Unable to resolve ${hostname}`);
                    }
                    console.log("DNS look up ended for: " + hostname);
                    cb(null, ips[0], 4);
                });
            };

As far as I know there is no DNS cache on linux based OS. What is causing such a big jump when replaced with dns resolve? Does it not use same dns servers as dns lookup ? Still 6 seconds is a long time as compared to 1.4 seconds for processing http requests

Mary123
  • 11
  • 4
  • I wouldn't be at all surprised if your "parallel queries" looked like a Denial of Service attack to some DNS servers...and perhaps they're responding accordingly... – paulsm4 Jul 08 '22 at 23:53
  • `If I understood it correctly, node http module does dns lookup using worker threads and hence number of lookups at a time are limited by libuv threads.` Is written where? A DNS lookup speed depends on the given server to perform the lookup against. https://nodejs.org/dist/latest-v16.x/docs/api/dns.html#dnsgetservers – Marc Jul 08 '22 at 23:55
  • @paulsm4 Thats not the case because they are all different domains. Also as i mentioned, when using default dns lookup it works fine – Mary123 Jul 09 '22 at 00:10
  • @Marc See "What code runs on Worker Pool" https://nodejs.org/en/docs/guides/dont-block-the-event-loop/ – Mary123 Jul 09 '22 at 00:17
  • Why do you suspect worker threads and DNS are limiting the performance of http request rate? – Matt Jul 09 '22 at 05:15
  • Also distributions using systemd-resolved will have a local cache. – Matt Jul 09 '22 at 05:18
  • If you got to the link above, it explains how number of DNS resolutions at a time is restricted to 4 when using dns.lookup(default number of libuv threads), so I was exploring using dns.resolve instead of default dns.lookup. Also see - https://httptoolkit.tech/blog/configuring-nodejs-dns/ – Mary123 Jul 11 '22 at 07:25

0 Answers0