How solve java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem in aws lambda?

Question

After setup the hadoop configuration I got java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.fs.LocalFileSystem even when org.apache.hadoop.fs.LocalFileSystem is present in the jar file .

Answer 1

Solution is to "remove" classloader from getClassByName function. E.g. create org.apache.hadoop.conf.Configuration with getClass overrided method, like:

Configuration hadoopConfig = new Configuration() {
    public Class<?> getClassByName(String name) throws ClassNotFoundException {
      return Class.forName(name);
    }

};

Notes

1. This is just work implementation. Original implementation uses Class.forName(name, true, classLoader); so new overrided code may not work if so add if statement to change logic obly for org.apache.hadoop.fs.LocalFileSystem
2. The error happened and fixed only for AWS lambda. In other environments (like true Hadoop cluster) this may not be an appropriate solution