1

I have been able to connect to my proxy using their source code which is like this with personal data omitted:

'use strict';
var http = require('http');
var port = process.env.PORT || 1337;
var url = require('url');
var dirlevel = '..';
var runPup = require('./mypuppeteerscript.js');
http.createServer(function (req, res) {
    var page = req.url;
     var request = require('request');

            request({
                uri: 'http://httpbin.org/get',
                proxy: 'http://<my api key>:@proxy.crawlera.com:8011'
            }, function callback(error, response, body) {
                console.log(body);
            });
            runPup.runPuppeteer(page);
            res.writeHead(200, { 'Content-Type': 'text/plain' });
            res.end('My App - Alpha\nProcessing File\n' + page);
}).listen(port);

The following is the block that was pasted from the Zyte website:

var request = require('request');

            request({
                uri: 'http://httpbin.org/get',
                proxy: 'http://<my api key>:@proxy.crawlera.com:8011'
            }, function callback(error, response, body) {
                console.log(body);
            });

Unfortunately, that does not result in every request from my Puppeteer script being routed through the proxy. How would I modify this code so that every request from runPup.runPuppeteer(page) is routed through the proxy?

My options are somewhat limited here because Zyte cannot be accessed via Puppeteer directly due to my Puppeteer version being greater than 1.17. To use Zyte in my Puppeteer code I would need to install Docker on my Windows Server. I have never used Docker and my time is limited.

WannabePuppetMaster
  • 123
  • 1
  • 7
  • Just to be clear, when you say that the proxy can't be accessed via Puppeteer directly, you mean that you've tried the suggested solution [here](https://stackoverflow.com/a/52778119/996081)? – cbr Apr 14 '21 at 03:14
  • Yes, that does not work because of some conflict between Zyte and every version of Puppeteer over 1.17, so I though maybe if there were a way to make the node app route all requests through a proxy that it would automatically include the Puppeteer requests since those are further down the vine. – WannabePuppetMaster Apr 14 '21 at 05:06
  • I tried using http-proxy also but it seems that the requests are still going directly from Puppeteer to the target URLs and not via the target specified in proxy.web – WannabePuppetMaster Apr 14 '21 at 05:14

0 Answers0