How to retry a request based on response headers using request and nodejs?

Question

I have a small nodejs script and I want to download a zip file:

const fs = requrie("fs");
const request = require("requestretry");

export class FileFetcher {
    public async fetchFile(url: string): Promise<string> {
        const fileName = "./download.zip";
        return new Promise<string>(resolve => {
            request(url)
                .pipe(fs.createWriteStream(fileName))
                .on("close", function () {
                    resolve(fileName);
                });
        });
    }
}

I am using the requestretry library, a wrapper around request, as the file may only exists in the future and hence will most likely fail for the first couple times.

Yet I have to adapt to my strangely behaving external endpoint. It always returns a 200 OK instead of an expected 404 Not Found.

Hence, the only way to determine is the Content-Length header in the response. If Content-Length: 0 then there is no file.

There seems to be different retry strategies in requestretry, yet they all seem to assume a well-beaved api, e.g. request.RetryStrategies.HTTPOrNetworkError. Since it gets a 200 it will never retry and "successfully" download an empty zip file.

I am not fixed on using requestretry, I am wondering about how to retry a request based on the response headers.

score 2 · Answer 1 · answered Sep 30 '16 at 00:16

2

You could define your own retry logic with retryStrategy, e.g.:

request({
  url: '...',
  retryStrategy: function(err, res, body) {
    return res.headers['Content-Length'] === 0;
  }
});

answered Sep 30 '16 at 00:16

a0viedo

199
1
11

score 0 · Accepted Answer · answered Oct 06 '16 at 16:30

I ended up ditching the requestretry library and went with request and implemented the retry mechanism myself.

One issue was to get the actual response body data. I expected it to be available through the response event, yet this only has the header.

The data can be fetched through multiple data event and resolving the promise on the end event.

Another gotcha was understanding how the request needs to be send as encoding needs to be null or the downloaded file is casted to a string and hence corrupt.

My downloader class looks like:

export class FileFetcher {
    public async downloadWithRetry(url: string, maxRetries: number, timeout: number): Promise<Buffer> {
        while (true) {
            try {
                const buffer = await this.fetchFile(url);
                return new Promise<Buffer>(resolve => {
                    resolve(buffer);
                });
            } catch (e) {
                maxRetries = maxRetries - 1;

                if (maxRetries <= 0) {
                    throw new Error("Too many requests");
                }

                console.warn(`No file at url:${url}, waiting ...`);
                await this.wait(timeout);
                console.log(`Continue.`);
            }
        }
    }

    private async wait(timeout: number): Promise<any> {
        timeout *= 1e3;
        console.log(timeout);
        return new Promise(resolve => {
            setTimeout(resolve, timeout);
        });
    }

    private async fetchFile(url: string): Promise <Buffer> {
        return new Promise<Buffer>((resolve, reject) => {
            let data = [];
            request.get({
                encoding: null,
                url: url,
            }).on("data", function (chunk) {
                data.push(chunk);
            }).on("response", function (response) {
                /**
                 * Server always returns 200 OK even if file does not exist yet. Hence checking for content-lenth head
                 */
                if (!(response.headers["content-length"] > 0)) {
                    reject("Empty response!");
                }
            }).on("end", function () {
                const body = Buffer.concat(data);
                if (body.length > 0) {
                    resolve(body);
                    return;
                }

                reject("Empty response");
            });
        });
    }
}

That buffer can be written via fs.writeFile(file, buffer).

Downside is not using the stream approach anymore as I wanted to pipe the data. Yet this approach at least fetches the file properly.

How to retry a request based on response headers using request and nodejs?

2 Answers2