why my code is returning 403 forbidden acess

Question

I am trying to make a http get request using java ,when i am executing the code ,i am getting 403 forbiddencode .Is there any way to get rid of that ?my code is

 import java.io.BufferedReader;
 import java.io.InputStreamReader;
 import java.net.*;


public class Http {
private final String USER_AGENT = "Mozilla/5.0";
public static void main(String [] args){

    Http http=new Http();
    try {
        http.get();
    } catch (Exception e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

}
private void get() throws Exception{
    URL ob=new URL("http://www.google.com/search?q=vamsi");
    HttpURLConnection con=(HttpURLConnection) ob.openConnection();
    con.setRequestMethod("GET");
    con.setRequestProperty("USER_AGENT", USER_AGENT);
    int responseCode=con.getResponseCode();
    System.out.println("Response code is "+responseCode);
    BufferedReader buf=new BufferedReader(new InputStreamReader(con.getInputStream()));
    String input;
    StringBuffer response =new StringBuffer();
    while((input=buf.readLine())!=null){
        response.append(input);

    }
    buf.close();

    System.out.println(response.toString());
}   


}

Can we assume you can successfully visit that URL from (a) a browser and (b) the command line using `curl` or `wget` ? — Stewart, Feb 02 '14 at 06:19

score 6 · Accepted Answer · edited May 23 '17 at 12:11

6

Google Search will deny your request if you are doing it outside of their API and Terms of Service.

$ wget http://www.google.com/search?q=vamsi
Resolving www.google.com (www.google.com)... 74.125.225.212, 74.125.225.210, 74.125.225.211, ...
Connecting to www.google.com (www.google.com)|74.125.225.212|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2014-02-01 23:21:08 ERROR 403: Forbidden.

They may be looking at your User-Agent header. If you add one that resembles a browser, it may work, though you are circumventing the Terms of Service and seeking unsupported behavior. (Sorry, I too tried this once.)

See also Why does Google Search return HTTP Error 403?

FYI, this is a common restriction of major search engines (Google, Bing, Yahoo). There are some that you can query programatically; you will have to use those.

edited May 23 '17 at 12:11

Community

1
1

answered Feb 02 '14 at 06:23

Paul Draper

78,542
46
206
285

2

*"... though you're really entering undefined behavior"* - And violating their ToS! **WARNING:** - Google would be within their rights to black-list your IP address for spoofing your User-Agent to side-step the ToS restriction. – Stephen C Feb 02 '14 at 06:29
I think you're right, I tried using a FF plugin called Poster and set the header: `User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.102 Safari/537.36` and got 200OK response – Nir Alfasi Feb 02 '14 at 06:29
@Paul: dude is there a way to spoof ? – Vamsi Pavan Mahesh Feb 02 '14 at 06:32
@VamsiPavanMahesh - dude. You do not want to do that. Seriously! – Stephen C Feb 02 '14 at 06:33
@StephenC: I mean ,i just want to know a method,if i can do that ..just for knowledge sake ! – Vamsi Pavan Mahesh Feb 02 '14 at 06:35
@VamsiPavanMahesh, yes, though spoofing by definition enters undefined or forbidden behavior, so the answer (if one were given) may not work two months from now. – Paul Draper Feb 02 '14 at 06:41

score 0 · Answer 2 · answered Feb 02 '14 at 06:44

0

This is the solution:

con.setRequestProperty("User-Agent", USER_AGENT);

answered Feb 02 '14 at 06:44

Alvin Bunk

7,621
3
29
45

You had "USER_AGENT" entered in your code, but instead should be "User-Agent"; That fixed the problem. – Alvin Bunk Feb 02 '14 at 06:45

why my code is returning 403 forbidden acess

2 Answers2