java.lang.NoSuchFieldError: IBM_JAVA

Question

I see many references to this question and I have followed posted in here

I am using Hadoop 2.4.1 and Flume 1.5.0.1. My flume-env.sh configuration is as below

FLUME_CLASSPATH="/var/lib/apache-flume-ng:lib/hadoop-core-1.2.0.jar:lib/hadoop-auth-2.4.1.jar:lib/hadoop-yarn-api-2.4.1.jar:lib/hadoop-mapreduce-client-jobclient-2.4.1.jar:lib/hadoop-mapreduce-client-core-2.4.1.jar:lib/hadoop-common-2.4.1.jar:lib/hadoop-annotations-2.4.1.jar"

with these jars I have added one more jar that is commons-configuration-1.6.jar available in lib of Flume. I am new to Flume and Hadoop.

Complete trace is as follows:

ERROR [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:149)  - Unhandled error
java.lang.NoSuchFieldError: IBM_JAVA
    at org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName(UserGroupInformation.java:337)
    at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:382)
    at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:553)
    at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:272)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

score 6 · Answer 1 · answered Jul 29 '15 at 03:02

6

The problem is caused by missing or conflicting dependencies:

Add hadoop-auth to your classpath
If the problem still arises, remove hadoop-core from your classpath. It is conflicting with hadoop-auth.

This will solve the problem.

answered Jul 29 '15 at 03:02

JayL

574
5
18

yes, this solved my problem. I had to add hadoop-auth and remove hadoop-core. – edge Dec 25 '16 at 14:50

Raghuveer · Accepted Answer · 2018-08-04T03:55:48.833

Finally i found an answer for this.

Copy the PlatformName class from hadoop-auth and custom compile it locally.

package org.apache.hadoop.util;

public class PlatformName {

    private static final String platformName = System.getProperty("os.name") + "-" + System.getProperty("os.arch") + "-" + System.getProperty("sun.arch.data.model");

    public static final String JAVA_VENDOR_NAME = System.getProperty("java.vendor");

    public static final boolean IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");

    public static String getPlatformName() {
        return platformName;
    }

    public static void main(String[] args) {
        System.out.println(platformName);
    }
}

Copy paste the class file in your hadoop-core. You should be up and running.

thanks to all

Solution is to add all the jars found in $HADOOP_HOME/share/ all libraries. By that you dont have to add any custom code at all — Raghuveer, Oct 13 '14 at 07:36

java.lang.NoSuchFieldError: IBM_JAVA

2 Answers2