Unhandled error java.lang.NoSuchMethodError while running flume for downloading twitter data

Question

When i run this command

hduser@ubuntu:/usr/local/flume$ bin/flume-ng agent --conf ./conf/ -f conf/twitterflume-agent.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent

the flume is getting started but after some time it throws me the exception which doesn't allow flume to download. I m getting the following error:

2015-10-31 10:18:32,152 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:40)] Creating instance of sink: HDFS, type: hdfs

2015-10-31 10:18:35,386 (conf-file-poller-0) [ERROR - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:149)] Unhandled error
java.lang.NoSuchMethodError: com.google.common.cache.CacheBuilder.refreshAfterWrite(JLjava/util/concurrent/TimeUnit;)Lcom/google/common/cache/CacheBuilder;
    at org.apache.hadoop.security.Groups.<init>(Groups.java:97)
    at org.apache.hadoop.security.Groups.<init>(Groups.java:74)
    at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:303)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:283)
    at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:260)
    at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:337)
    at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:331)
    at org.apache.flume.sink.hdfs.HDFSEventSink.authenticate(HDFSEventSink.java:491)
    at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:240)
    at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

After this error, it trys to check but it failed to download the data.

2015-10-31 10:28:05,446 (conf-file-poller-0) [DEBUG - org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:126)] Checking file:conf/twitterflume-agent.conf for changes
[enter image description here][1]

I'm a fresher to Hadoop and Flume.

score 1 · Answer 1 · answered Nov 02 '15 at 07:31

Seems like a compile and run-time libraries incompatibility. There's a difference between the libraries present when compiling the code and when running the code.

Check if you are providing the correct classpath when running flume code. On the node where the agent is running, run ps aux|grep flume to check the classpath.

Also note that if you changed the Twitter code and bumped up the libraries, it will cause this issue because Flume also comes with its own Twitter source. In that case, either delete the Twitter jars or better yet remove the Flume's twitter jars from the classpath.

Unhandled error java.lang.NoSuchMethodError while running flume for downloading twitter data

1 Answers1