1

I've tried a number of ways of using Zyte (formally Crawerla) proxies with Selenium.

They provide 1- API key (username) 2- Proxy url/port.

No password is needed.

What I have tried...

    ChromeOptions options = new ChromeOptions();
    var proxy = new Proxy();
    proxy.Kind = ProxyKind.Manual;
    proxy.IsAutoDetect = false;
    proxy.SocksUserName = "<<API KEY>>";
    proxy.SocksPassword = "";
    proxy.HttpProxy =
    proxy.SslProxy = "proxy.crawlera.com:8011";
    options.Proxy = proxy;
    IWebDriver driver = new ChromeDriver(options);

Which, when selenium loads produces this: enter image description here

Funnily enough, if I manually add the username (API Key) it will indeed load, but this defeats the purpose of automation.

The second method I tried was:

    ChromeOptions options = new ChromeOptions();
    options.AddArguments("--proxy-server=<API KEY>::proxy.crawlera.com:8011");
    options.AddArguments("--log-level=OFF");
    IWebDriver driver = new ChromeDriver(options);

I used :: as the password is blank.

And the error to this method is:

[46784:44492:0219/015119.757:ERROR:validation_errors.cc(87)] Invalid message: VALIDATION_ERROR_DESERIALIZATION_FAILED

I guess Zyte/Crawerla knowledge isn't really needed, it's more how to provide selenium with a username, but no password, and have it use the proxy successfully.

Does anybody have any idea? (examples appreciated)

MattHodson
  • 736
  • 7
  • 22

1 Answers1

0

you can use this Crawlera-Headless-Proxy https://github.com/zytedata/zyte-smartproxy-headless-proxy when using a headless browser. This will act as a MITM proxy and will handle the Authentication of SPM as well. So you will not need to mention the Crawlera/SmartProxyManager Authentication within your C# Selenium Script.

Crawlera-headless-proxy can be used as a standalone binary like this

crawlera-headless-proxy -c config.toml

You can mention the API Key, port number, and other configs in the config.toml file.

It will start the MITM server at the localhost:3128. In your C# you need to mention the proxy host/port should be localhost/127.0.0.1 and port as 3128.

The requests from your C# headless script needs to be sent to Crawlera-Headless-Proxy and then Crawlera-Headless-Proxy will send the request to the SmartProxyManager/Crawlera.

So your code may look like this for ex:

ChromeOptions options = new ChromeOptions();
var proxy = new Proxy();
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.HttpProxy =
proxy.SslProxy = "127.0.0.1:3128";
options.Proxy = proxy;
IWebDriver driver = new ChromeDriver(options);
mukthy
  • 1
  • 1