java Api for BGP Ranking

Question

i am trying to hit bgp ranking url but am getting html page as output , i want to get the json data in that hmtl page. how can we get the json data.

This is the controller and below that i have given the service implementation. can any body find out the issue

this is the BGP Ranking Url :https://bgp.he.net

Controller

@RestController
@RequestMapping(value = "/api/bgp")
public class BGPController {

    @Autowired
    BGPService bgpService;

    @RequestMapping(value = "/getInfoById/{query}", method = RequestMethod.POST)
    protected @ResponseBody Map<String, Object> getAllJobs(@PathVariable(value = "query") String query) {
    Map<String, Object> dataMap = new HashMap<String, Object>();
    try {
        dataMap.put("status", true);
        dataMap.put("result", bgpService.getRanking(query));
    } catch (Exception e) {
        dataMap.put("status", false);
        dataMap.put("reason", e.getMessage());
        System.err.println("exception at bgp: ");
        e.printStackTrace();
    }
    System.out.println("dataMap: " + dataMap.toString());
    return dataMap;
    }
}

Service Implementation

@Service
@Component
public class BGPServiceImpl implements BGPService {

    static CloseableHttpClient httpClient = HttpClients.createDefault();
    HttpGet getRequest = null;
    CloseableHttpResponse response = null;
    BufferedReader br = null;
    String stringJson = null;
    StringEntity stringEntity = null;
    String output = null;

    String URL = "https://bgp.he.net/";

    @Override
    public Map<String, Object> getRanking(String query) {

        Map<String, Object> dataMap = new HashMap<>();
        try {
            getRequest = new HttpGet(URL  + query);
            response = httpClient.execute(getRequest);
            br = new BufferedReader(new InputStreamReader((response.getEntity().getContent())));
            String responseContent = "";
            while ((output = br.readLine()) != null) {
            responseContent += output;
            }
            if (response.getStatusLine().getStatusCode() == 200) {
            dataMap.put("status", "success");
            dataMap.put("rawData", responseContent);
            } else {
            dataMap.put("status", "failure");
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
            br.close();
            } catch (Exception e) {
            e.printStackTrace();
            }
        }
        System.out.println("dataMap: " + dataMap);
        return dataMap;
    }

}

OneCricketeer · Answer 1 · 2019-02-08T06:49:07.603

0

but am getting html page as output , i want to get the json data in that hmtl page

You need to parse the HTML Jsoup would be useful library for this. And then you must create your own JSON.

However, that search box on the site doesn't seem to be returning any results, so I'm not sure your code is going to work without some extra efforts, like using Selenium instead

Use the terminal, not a browser to test your query first.

$ curl -sL 'https://bgp.he.net/query'

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /query
on this server.</p>
<hr>
<address>Apache/2.4.7 (Ubuntu) Server at bgp.he.net Port 443</address>
</body></html>

edited Feb 08 '19 at 06:49

answered Feb 08 '19 at 06:13

OneCricketeer

179,855
19
132
245

how can we parse using json – khaja mohiddin Feb 08 '19 at 06:18
That's too broad to answer here. I trust you can search for that elsewhere – OneCricketeer Feb 08 '19 at 06:19
can you share me any link for reference – khaja mohiddin Feb 08 '19 at 06:22
https://stackoverflow.com/questions/tagged/java+json?sort=votes&pageSize=15 – OneCricketeer Feb 08 '19 at 06:24
i don't have knowledge on jsoup can you please help me out – khaja mohiddin Feb 08 '19 at 06:39
I would suggest you try something, then post a new question with your efforts. – OneCricketeer Feb 08 '19 at 06:48

score 0 · Answer 2 · answered Feb 08 '19 at 08:32

I think there are three main steps.

First, Request html page with https connection whichever tools you know like httpsurlconnection class etc.

Second, you have to find your keyword in the html page then parse it with the key using jsoup library.

Finally, you can use json library and get a json string with a litte effort.

For example, in the first page of the site

I can find a keyword, 'tabdata' after searching the source code of the html.

The rest process goes smoothly.

Here is my example for you.

First, I have to skip this step because you already have your own solution.

Second, A JSoupParser class, which is extracting contents with the keyword

static class JSoupParser {
    public String getWelcomeBGP(final String htmlcontents) {
        StringBuilder stbuld = new StringBuilder();
        Document doc = Jsoup.parseBodyFragment(htmlcontents);

        for (Element div : doc.select("div")) {
            int i = 0; 
            if(div.className().equals("tabdata"))
            {
                for (Element subdiv : div.select("div")) {
                    if(i != 0)
                        if(!subdiv.text().equals(""))
                            stbuld.append(subdiv.text()).append("\n");
                    i++;
                }
            }
        }

        return stbuld.toString();
    }
}

finally, A JsonParser class to generate a json string from the contents.

static class JsonParser {
    public JSONObject getWelcomeBGP(final String contents) throws IOException {
        BufferedReader breader = new BufferedReader(new StringReader(contents));
        String line= null;
        JSONObject jobj = new JSONObject();
        int id = 0;
        while((line = breader.readLine()) != null)
        {
            jobj.put("A" + id++, line);
        }

        return jobj;
    }

}

Here is my main method.

import org.json.JSONObject;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;

public static void main(String[] args) {
    // first, request html contents from the site.
    HttpsClientWithoutValidation htmlContents = new HttpsClientWithoutValidation();
    final String url = "https://bgp.he.net";
    String response = htmlContents.requestHtmlContents(url);

    // second, request html contents from the site.
    JSoupParser htmlparser = new JSoupParser();
    String contents = htmlparser.getWelcomeBGP(response);

    // finally, make your own json string or object whatever.
    JSONObject jobj = null;
    JsonParser jsonparser = new JsonParser();
    try {
        jobj = jsonparser.getWelcomeBGP(contents);
        System.out.println(jobj.toString());
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
}

Have a good day.

java Api for BGP Ranking

2 Answers2