HTTP 403 from Java app (but not web browser), despite User-Agent set

Question

I'm using ROME to poll and aggregate RSS feeds that refreshes every twenty minutes. To circumvent the possible lack of a User-Agent, I artificially added the one I retrived from my installation of Chrome. The relevant code bit looks like this:

URLConnection connection = new URL(feed.getFeedUrl()).openConnection();
connection.setRequestProperty("User-Agent",
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36");

SyndFeedInput input = new SyndFeedInput();
XmlReader reader = new XmlReader(connection.getInputStream(),
        "text/html; charset=UTF-8", true);
SyndFeed syndFeed = input.build(reader);

(feed.getFeedUrl() returns the URL for a feed as a String). This works for most feeds I'm polling, but does not for the one located at https://eurovoix.com/feed/, which returns HTTP error code 403 ("Forbidden") upon response. The feed works fine when called from the browser. What could be causing this?

Edit: Attempting to use the solution from this thread -- adding CookieHandler.setDefault(new CookieManager(null, CookiePolicy.ACCEPT_ALL)); -- does not resolve the issue, unfortunately.

Instead of using URLConnection, use any RESTClient like jersey-client, spring rest template or apache http-client. — Sambit, Jun 02 '19 at 17:09
Unfortunately, this too does not solve my issue. I tested Jersey Client (I was already using Jersey Container for the server application to have a REST interface), and the generated response was also 403. — , Jun 02 '19 at 17:32

score 0 · Answer 1 · answered Jun 02 '19 at 18:42

As per the above discussion, I used Jersey rest client and I have tested it is working fine for me. You can also try. I used the following jar file.

jersey-client version 1.8

If you are using maven, you can include the following dependency in pom.xml.

<dependency>
    <groupId>com.sun.jersey</groupId>
    <artifactId>jersey-client</artifactId>
    <version>1.8</version>
</dependency>

I provide below the code, you can test and verify.

import com.sun.jersey.api.client.Client;
import com.sun.jersey.api.client.ClientResponse;
import com.sun.jersey.api.client.WebResource;

public class TestGetCallByJersey {
  public static void main(String[] args) {
    String resourceUri = "https://eurovoix.com/feed";
    try {
      Client client = Client.create();
      WebResource webResource = client.resource(resourceUri);
      ClientResponse response =
          webResource
              .accept("application/xml")
              .header("User-Agent", "Mozilla/5.0")
              .get(ClientResponse.class);

      System.out.println("response status = " + response.getStatus());
      String result = response.getEntity(String.class);
      System.out.println("Output from api call .... \n" + result);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

com.sun.jersey.jersey-client is outdated, org.glassfish.jersey.core.jersey-client should be used instead. Either way, your code closely resembles the one I wrote yesterday, after which I replied that it did not work. Your snippet worked for some reason, so I went back and checked again, and my old code works again for that website. I'm not sure why, but this might have been a server-side issue. Thanks. — , Jun 03 '19 at 16:11

HTTP 403 from Java app (but not web browser), despite User-Agent set

1 Answers1