Using proxy to make request results in bad request (400) error code

Question

I'm using node-fetch and https-proxy-agent to make a request using a proxy, however, I get a 400 error code from the site I'm scraping only when I send the agent, without it, everything works fine.

import fetch from 'node-fetch';
import Proxy from 'https-proxy-agent';

const ip = PROXIES[Math.floor(Math.random() * PROXIES.length)]; // PROXIES is a list of ips
const proxyAgent = Proxy(`http://${ip}`);

fetch(url, {
    agent: proxyAgent,
    headers: {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.72 Safari/537.36'
    }
}).then(res => res.text()).then(console.log)

This results in a 400 error code like so:

I have absolutely no idea why this is happening. If you want to reproduce the issue, I'm scraping https://azlyrics.com. Please let me know what is wrong.

score 1 · Accepted Answer · answered Jun 05 '22 at 13:54

1

The issue has been fixed. I did not notice I was making a request to a https site with a http proxy. The site was using https protocol but the proxies were http only. Changing to https proxies works. Thank you.

answered Jun 05 '22 at 13:54

FC5570

163
4
12

Using proxy to make request results in bad request (400) error code

1 Answers1