Jackson deserialisation/TypeReference for dynamically loaded pojo class

Question

I have a requirement to get JSON input Pojo instance and I am using Jackson 2 library and below readValue method could deserialise using typeReferencing :

POJO_ClassName p = mapper.readValue(new TypeReference< POJO_ClassName >() {});

But the problem is that as POJO is created and loaded at runtime dynamically, how do I get JSON to POJO instance/object as I do not have fully qualified class (POJO_ClassName) name for above statement?

Note: I use jsonSchema2pojo library to generate POJO classes at runtime.

Here is code snippet, I am using to generate POJO for JSON at runtime and trying

  String classPath="com.EnrichmentService.Thread72"; 
     String classLocation = System.getProperty("user.dir")
                         + "/src/main/java"; JCodeModel codeModel = new JCodeModel();

     final RuleFactory ruleFactory = new RuleFactory(config,
                         new Jackson2Annotator(config), new SchemaStore());

     final SchemaMapper mapperSchema = new SchemaMapper(ruleFactory,
                         new SchemaGenerator());

     mapperSchema.generate(codeModel, "EsRootDoc",classPath, json);

     codeModel.build(new File(classLocation));  // generates pojo classes

     // Till above jsonSchema2Pojo pojo generation all Good !!
      // EsRootDoc instance is needed for further drools drl validations.

     com.EnrichmentService.Thread72.EsRootDoc p = mapper.readValue(new TypeReference<com.EnrichmentService.Thread72.EsRootDoc>() {}); 
// see alternative way as well in my 24Aug17 edit at the end of this question

But as com.EnrichmentService.Thread72.EsRootDoc has yet not been generated compiler would error to class not Found.

Main Points:

1) Same Pojo classes generated at run time iteratively but with different properties as JSON input changes each time.

2) Even tried Object pojo =mapper.readValue(json,Class.forName("com.EnrichmentService.Thread72.EsRootDoc")); as class.forName does not replace an existing class!

Edit 24 Aug17 - Here is my custom class loader :

Note: Indexer is class which load dynamic EsRootDoc/POJO class at run time.

 static class TestClassLoader extends ClassLoader {
            @Override
            public Class<?> loadClass(String name) throws ClassNotFoundException {
                if (name.equals("com.EnrichmentService.Thread72.EsRootDoc")) {
                    try {
                        InputStream is = Indexer.class.getClassLoader().getResourceAsStream("com/EnrichmentService/Thread72/EsRootDoc.class");
                        byte[] buf = new byte[is.available()];
                        int len = is.read(buf);

                        Class<?> c=defineClass(name, buf, 0, len);
                        resolveClass(c);
                        return c;


                    } catch (IOException e) {
                        throw new ClassNotFoundException("", e);
                    }
                }
                return getParent().loadClass(name);
            }
        }

I have tried using above TestClassLoader custom class loader as an alternative way is like this :

Class cls = new      TestClassLoader().loadClass("com.EnrichmentService.Thread72.EsRootDoc");
    Object obj = cls.newInstance();
    cls.getMethod("getCrawlerSource").invoke(obj);
    p=mapper.readValue(json, cls);  // but here i am getting the same deserialization exception as earlier.

Referred an old answer@ How to replace classes in a running application in java ?

Edit2: 24Aug17 Exception being faced stackTrace is here: https://pastebin.com/ckCu2uWx

Answer 1

you have found, you can only use TypeReference with types that are known at compile time (without some very tricky meta-programming).

However, there are lots of alternative overloads to readValue which do not require a TypeReference, to allow for cases like yours where TypeReference is impractical.

I think you can use readValue(... , Class<T> valueType)

If you have a special Classloader for these late-compiled classes, then you can get a Class instance from that and pass it in, for example:

ClassLoader dynamicallyCompiledPojoLoader = ...;
Class<?> pojoClass = dynamicallyCompiledPojoLoader.loadClass("...");

return mapper.readValue(..., pojoClass);

See also com.fasterxml.jackson.databind.type.TypeFactory for specifying parameterised generic types without using TypeReference

Update after "Edit2: 24Aug17 Exception being faced stackTrace is here"

Your current exception ( https://pastebin.com/ckCu2uWx ) is not a class loader issue, but a JSON schema mismatch issue.

The relevant part of the exception message is:

Can not deserialize instance of java.util.ArrayList out of START_OBJECT token
...
through reference chain: com.EnrichmentService.Thread72.EsRootDoc["category"]->java.util.ArrayList[0]->com.EnrichmentService.Thread72.Category["crawler"])

So Jackson is unhappy that the "crawler" field in the JSON is an object, i.e. starts with "{", but the Java property "crawler" is an ArrayList, i.e. should start with "["

I don't know why the POJO structure of Thread72.Category is wrong here, but it doesn't look like a classloader problem.

Update after commenting that the POJO class changes with each request

You have said that 1) the POJO class structure varies with each request and 2) you want to use the same classname for the POJO each time.

See Java - how to load different versions of the same class?

You'll need to use a new Classloader for each request, as classes get cached by Classloaders. The code you have posted so far suggests that you are using a single classloader and hoping for a new load of the "Category" class on each request, which won't work.

You have said:

I need to generate new classes each time for a drools based elasticsearch document re-indexing work and this drools setup needs pojo/Object type instances.thanks

... but I think you should look into using a Map or similar input to Drools rather than a reflection based POJO, given that you don't know the structure in advance. As you have found here, this is a poor fit for the class / classloader abstraction.

See e.g.

• https://groups.google.com/forum/#!topic/drools-usage/CbuSO-V-w_g
• Convert Java POJO to Drools DRL and vice versa
• https://groups.google.com/forum/#!topic/drools-usage/0BIXF3Tg5pw

Answer 2

Imo there are two approaches to solve that:

1. create & compile the classes at comile time (e.g. with maven and jaxb)

or

2. you do something like that:

   String className = "com.EnrichmentService.Thread72.EsRootDoc";
   Class<?> clazz = Class.forName(className);
   Object object = clazz.getConstructor().newInstance();
   Object p = mapper.readValue(json, object.getClass());
   

If that code fails before mapper.readValue() you have another problem (my guess would be with classloading).

Even better would be something with generics:

    String className = "com.EnrichmentService.Thread72.EsRootDoc";
    Class<?> clazz = Class.forName(className);
    // cannot use dynamically created classes in a static way, just to 
    // show the point
    // com.EnrichmentService.Thread72.EsRootDoc p = 
    //     getObjectFromMessageString(json, clazz);
    Object p = getObjectFromString(json, clazz);

    public static <T> T getObjectFromString(String json, Class<T> clazz) {
        return mapper.readValue(json, clazz);
    }

Edit:

I wrote some example code which compiles a class on runtime and then tries to convert to an object of said compiled class. The output was as I expected:

import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;

import javax.tools.JavaCompiler;
import javax.tools.ToolProvider;

import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectMapper;

public class JackonCustomClassTest {
    public static String CLASS_NAME = "EsRootDoc";
    public static String PACKAGE_NAME = "com.EnrichmentService.Thread72";
    public static String CANONICAL_NAME = PACKAGE_NAME + "." + CLASS_NAME;

    public static void main(String args[]) throws Exception {
        JackonCustomClassTest mtc = new JackonCustomClassTest();
        Class<?> c = null;
        String source = null;
        // compile class for the first time
        source = "package "+PACKAGE_NAME+"; public class "+CLASS_NAME+" { public "+CLASS_NAME+"() { }; public String toString() { return \"Name: not existing\" + \" - className: \" + getClass().getCanonicalName(); }; }";
        c = mtc.compileClass(CANONICAL_NAME, source);

        System.out.println("class test: " + c.newInstance().toString());

        // compile class for the second time
        source = "package "+PACKAGE_NAME+"; public class "+CLASS_NAME+" { private String name; public "+CLASS_NAME+"() { }; public String getName() { return name; }; public void setName(String name) { this.name = name; }; public String toString() { return \"Name: \" + name + \" - className: \" + getClass().getCanonicalName(); }; }";
        c = mtc.compileClass(CANONICAL_NAME, source);

        System.out.println("class test: " + c.newInstance().toString());

        mtc.runJackson(c);
    }

    private void runJackson(Class<?> clazz) throws JsonParseException, JsonMappingException, IOException {
        ObjectMapper m = new ObjectMapper();
        String string = "{ \"name\": \"asdf\" }";
        Object o = m.readValue(string, clazz);
        System.out.println("result of conversion: " + o); // Should print "Name: asdf"
    }

    public Class<?> compileClass(String fullyQualifiedClassName, String source) throws Exception {
        // Save source in .java file.
        File root = new java.io.File( "./target/test-classes/" );
        File sourceFile = new File(root, fullyQualifiedClassName.replace(".", "/") + ".java");
        sourceFile.getParentFile().mkdirs();
        Files.write(sourceFile.toPath(), source.getBytes(StandardCharsets.UTF_8));

        // Compile source file.
        JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
        compiler.run(null, null, null, sourceFile.getPath());

        // Load and instantiate compiled class.
        //          URLClassLoader classLoader = URLClassLoader.newInstance(new URL[] { root.toURI().toURL() });
        //          Class<?> cls = Class.forName(fullyQualifiedClassName, true, classLoader);
        Class<?> cls = new TestClassLoader().loadClass(fullyQualifiedClassName);
        return cls;
    }

    static class TestClassLoader extends ClassLoader {
        @Override
        public Class<?> loadClass(String name) throws ClassNotFoundException {
            if (name.startsWith(PACKAGE_NAME)) {
                try {
                    InputStream is = this.getClass().getClassLoader()
                            .getResourceAsStream(name.replace(".",  "/") + ".class");
                    byte[] buf = new byte[is.available()];
                    int len = is.read(buf);

                    Class<?> c = defineClass(name, buf, 0, len);
                    resolveClass(c);
                    return c;

                } catch (IOException e) {
                    throw new ClassNotFoundException("", e);
                }
            }
            return getParent().loadClass(name);
        }
    }
}

Edit 2:

Updated the code to try your TestClassLoader class - still get the correct (updated) version of the class.

Answer 3

But as com.EnrichmentService.Thread72.EsRootDoc has yet not been generated compiler would error to class not Found.

Yes, of course. If you don't have class in your JVM, you can't load them.

com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize

Please, give all stacktrace.

1) Same Pojo classes generated at run time iteratively but with different properties as JSON input changes each time.

Why you don't use Maps? Why you don't use one big class with all fields (and some of them will be nulls)?

Yes, the EsRootDoc class gets generated iteratively at run time and class changes as well as per input json changes in each iteration

If you do this in multiple threads, just synchronized them, for example:

final String ConcurrentHashMap<String, Class> dynamicClasses = new ConcurrentHashMap();

Object doInThreadOne(String json, String className) {
    return mapper.readObject(json, dynamicClasses.get(className))

void doInAnotherThread(String className) {
    dynamicClasses.put(className, reload(className));
}

If you need more strong consistency, then you can use synchronization by class name:

static final String className = "com.EnrichmentService.Thread72.EsRootDoc";

final String Map<String, Class> dynamicClasses = new HashMap();

Object doInThreadOne(String json) {
    synchronized(className) {
        return mapper.readObject(json, dynamicClasses.get(className))
    }

void doInAnotherThread(String className) {
    synchronized(className) {
        dynamicClasses.put(className, reload(className));
    }
}

Method Class.forName uses class loader of caller. You use different classloaders, it could be reason. There are overloads, where you can pass classloader.

Update for stackrace and additional information.

You should add @JsonIgnoreProperties(ignoreUnknown = true) to Crawler, because it has field subCategory, which is not present in Pojo.