how to change the hadoop sequence file value to jackson parser?

Question

I have one problem which I really don't know what to do. I have a Hadoop sequence file containing the links of the webpage. Each entry of the Hadoop sequence file, the key will be the URL of one webpage and the value would be its attributes and its links. The value actually is a Json format. I want to read all of the sequence file and pass the value to jackson parser to get the links, however it always failed. Here is my code:

    Configuration conf = new Configuration();
    Path seqFilePath = new Path("metadata-00000");

    SequenceFile.Reader reader = new SequenceFile.Reader(conf, 
            Reader.file(seqFilePath));
    Writable key = (Writable)ReflectionUtils.newInstance(reader.getKeyClass(), conf); 
    Writable value = (Writable)ReflectionUtils.newInstance(reader.getValueClass(), conf);

    int count = 0;
    while(reader.next(key, value)) {

        System.out.println(value.toString());
        JsonParser jsonParser = new JsonFactory().createJsonParser(value.toString());
        while(jsonParser.nextToken() != JsonToken.END_OBJECT) {
            String name = jsonParser.getCurrentName();
            if(name.equals("server_ip")) {
                System.out.println(jsonParser.getValueAsString());
            }

            if(name.equals("links")) {
                while(jsonParser.nextToken() != JsonToken.END_ARRAY) {
                    String attribute = jsonParser.getCurrentName();

                    if(attribute.equals("href")) {
                        System.out.println(jsonParser.getValueAsString());
                    }                   
                }
            }
        }

        long position = reader.getPosition();
        System.out.println(position);
        count++;

        if(count == 5) {
            break;
        }
    }

The file "metadata-00000" is the original Hadoop sequence file. As you can see, the value is actually in json format and I want to analyze it in Jackson parser. However, this line always failed:

JsonParser jsonParser = new JsonFactory().createJsonParser(value.toString());

the exception is:

Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonFactory
    at WebParser.ParserTest.main(ParserTest.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonFactory
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    ... 6 more

So how should I deal with it? How can I transfer the Writable value to json parser? Thanks!

score 0 · Answer 1 · answered Sep 19 '14 at 15:43

0

Your app cannot file class: com/fasterxml/jackson/core/JsonFactory

Verify if have jackson-core jar in your class path.

Or try adding this to your pom.xml file (if you are using maven as a build tool):

<dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-core</artifactId>
    <version>2.4.2</version>
</dependency>

answered Sep 19 '14 at 15:43

FazoM

4,777
6
43
61

Hi, I do contain the jackson-core.jar. This application is I make a jar file and run it on Hadoop. Does it mean I didn't include the jackson-core.jar while making the jar file? How should I deal with it? Thanks! – user2970089 Sep 19 '14 at 16:03
Yes, you didn't include it. What build tool are you using? (how do you create that jar?) There are different ways to include it, depending on your build tool. I have given you solution in my answer for maven. – FazoM Sep 19 '14 at 16:26
Hi,I download the jackson jar and included in the project. I didn't use maven. So in terms of this, how should I include the external jar in order to make it? Thanks! – user2970089 Sep 19 '14 at 17:00
Hi, I've already know how to include external jar, but there is still one problem. The Json parser seems to have nothing. I still don't know if I can pass the writable.toString() into the json parser. how should I deal with it? – user2970089 Sep 19 '14 at 17:36

how to change the hadoop sequence file value to jackson parser?

1 Answers1