0

I have a scraper that loops indefinitely, and I want to change the proxy on the client in case there is something wrong like an IP Ban (on 403 status code) or if I just want to change the proxy after each loop to avoid the bans completely.

The problem with this, specially the later case, is that eventually the program crashes when it reaches the maximum connections allowed (java.net.SocketException: No buffer space available (maximum connections reached?)).

Is there a way to change the HTTP Client's proxy without building a new one?

Adrián Montañes
  • 21
  • 4

1 Answers1

3

Is there a way to change the HTTP Client's proxy without building a new one?

I assume that you are talking about the JDK 11 (and later) java.net.http.HttpClient API.

The javadoc says this:

An HttpClient can be used to send requests and retrieve their responses. An HttpClient is created through a builder. The builder can be used to configure per-client state, like: the preferred protocol version ( HTTP/1.1 or HTTP/2 ), whether to follow redirects, a proxy, an authenticator, etc. Once built, an HttpClient is immutable, and can be used to send multiple requests.

My reading of that is that you cannot change the proxy; i.e. you cannot replace the client's ProxySelector object.

However, the ProxySelector (javadoc) is an abstract class, and you could create your own custom subclass that allows you to round-robin through a number of different proxies.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216