Writing a script in Java that regularly requests data from a website

Question

I'm working on a project that requires me to use information on border wait times provided by the Canadian Border Patrol on their website to construct a visual representation of wait time distribution.

I'm trying to find a way to have a Java script regularly check the website, and extract the information at a few different border stations (not all of them). I suppose I would use XPath to get me the specific stations, but how do I load up the webpage on a regular basis?

(P.S. I know they have a Twitter account too now, but they update it once a day and more specifically I'd like to learn how to work with websites and XPATH)

basically you have to implement AJAX with a timer code. It will very easy to implement using jQuery library. AJAX will work for you if have API access of the Website concerned. Or else you have to build some backend solution to scrape the data from that website — Harshith J.V., Feb 13 '13 at 05:34
You mean Javascript, not Java, right? If so please fix the tag. — Jim Garrison, Feb 13 '13 at 07:10

score 4 · Answer 1 · answered Feb 28 '13 at 00:34

4

Use the URL in Java. Create the URL and then use its method .openConnection() to start reading from the website.

import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;


public class webVisitor {


    public static void main(String[] args) {

        URL url;

        try {

            url = new URL("http://seinfeldaudio.com");
            URLConnection conn = url.openConnection();

            BufferedReader buffRead = new BufferedReader(new InputStreamReader(conn.getInputStream()));

            String inputLine = "";

            while (inputLine != null){
                inputLine = buffRead.readLine();
                System.out.println(inputLine);
            }


        }
        catch (Exception e){

        }

    }

}

More info here: http://www.mkyong.com/java/how-to-get-url-content-in-java/

answered Feb 28 '13 at 00:34

CodyBugstein

21,984
61
207
363

1

So you answered your own question and selected that as an answer ? – happybuddha Mar 05 '13 at 20:40
Anyway, you will also require some sort of push mechanism to update the UI with the new data. You may want to look at Comet push notifications for the same. – happybuddha Mar 05 '13 at 20:41
@happybuddha Ya basically. It was taking a long time to get an answer, so I took out a book and figured most of it out. Stackoverflow welcomes OP's answering their own questions btw – CodyBugstein Mar 06 '13 at 22:08
28 February i gave the exact answer for your question OP, i have even tested it and parsed the data, your answer is a generic answer in how to read the html data and print them, you have asked for XPATH parsing, but ok your question your answer :) – Thomas Sofras Mar 12 '13 at 10:33
@ThomasSofras I really do appreciate your answer, but I accepted mine since I had already done the research and posted my answer before yours was up. Nonetheless, out of appreciation, and because your answer is in fact better I will switch my accept :) – CodyBugstein Mar 12 '13 at 13:27
Really appreciate your stance :) – Thomas Sofras Mar 14 '13 at 10:44

score 4 · Accepted Answer · answered Feb 28 '13 at 10:16

Ok i had a little time off today at job and thought to give a help and write it for you. Excuse me for any mistakes it's the first time i parsed a site, i made a little research and decided to use jSoup for this.

Ok this code will parse the table and system out the 3 columns with the values, you can alter the code and build it within your needs :)

You have to download the jsoup jar Download jSoup


import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.URL;
import java.net.URLConnection;
import java.util.Iterator;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;


/**
 * 
 */
public class ParseWithJsoup{


    public static void main(String[] args) {

        URL url;

        try {

            url = new URL("http://www.cbsa-asfc.gc.ca/bwt-taf/menu-eng.html");
            URLConnection conn = url.openConnection();

            BufferedReader buffRead = new BufferedReader(new InputStreamReader(conn.getInputStream()));
            StringBuffer buffer = new StringBuffer("");

            String inputLine = "";

            // Append the site in a buffer
            while (inputLine != null){
                inputLine = buffRead.readLine();
                buffer.append(inputLine);
            }

            Document doc = Jsoup.parse(buffer.toString());

            // Parse the table
            Element table = doc.select("table[class=bwt]").first();

            //Office elements iterator
            Iterator<Element> officeElements = table.select("td[headers=Office]").iterator();

            //Commercial Flow iterator
            Iterator<Element> comElements = table.select("td[headers=Com ComCanada]").iterator();

            //Travellers Flow iterator
            Iterator<Element> travElements = table.select("td[headers=Trav TravCanada]").iterator();


            // Iterate all elements through first element row for all columns
            while(officeElements.hasNext()){            
                System.out.println("Office: " + officeElements.next().text());
                System.out.println("Commercial Flow: " + comElements.next().text());
                System.out.println("Travellers Flow: " + travElements.next().text());
            }

        }
        catch (Exception e){
            System.out.println("Exc:"+e.getMessage());
        }
    }


}

`

score 0 · Answer 3 · edited May 23 '17 at 11:50

0

Looks like Ajax using setInterval("function()",x)

Refer to this question - Repeat jQuery ajax call

I've not got around to learning node.js but this loos like something that would be well suited to it

edited May 23 '17 at 11:50

Community

1
1

answered Feb 13 '13 at 05:36

mr mojo risin

555
4
15

score 0 · Answer 4 · answered Mar 06 '13 at 16:12

0

Use the DWR (Easy Ajax for Java) ,call the DWR method from your java script by setting the time interval as

setInterval(DWR function here , millisec, lang)

In Java method use java.net.URL Class to read and parse the content as required.

answered Mar 06 '13 at 16:12

VKPRO

159
1
3
8

Writing a script in Java that regularly requests data from a website

4 Answers4