0

here moved snapshot jar in lib foldergetting this error after running flume command, i.e

user@ubuntu:~/hadoop_store/apache-flume-1.4.0-bin/bin$ ./flume-ng agent -n TwitterAgent -c conf -f ../conf/twitter1.conf

also tried

user@ubuntu:~/hadoop_store/apache-flume-1.4.0-bin/bin$ ./flume-ng agent -n TwitterAgent -c conf -f /home/user/hadoop_store/apache-flume-1.4.0-bin/conf/twitter1.conf

cause in one of similar question it was mentioned such, here is what exactly its saying

16/12/08 04:41:26 INFO source.DefaultSourceFactory: Creating instance of source Twitter, type org.apache.flume.source.twitter.TwitterSource
16/12/08 04:41:26 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: org.apache.flume.source.twitter.TwitterSource, class: org.apache.flume.source.twitter.TwitterSource
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
    at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: org.apache.flume.source.twitter.TwitterSource
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:195)
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)
    ... 11 more

Here is my twitter1.conf file,is there any error in this as such

# Naming the components on the current agent. 
TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel 
TwitterAgent.sinks = HDFS

# Describing/Configuring the source 
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = ###
TwitterAgent.sources.Twitter.consumerSecret = ###
TwitterAgent.sources.Twitter.accessToken = ###
TwitterAgent.sources.Twitter.accessTokenSecret = ###
TwitterAgent.sources.Twitter.keywords = lalalala

# Describing/Configuring the sink 

TwitterAgent.sinks.HDFS.type = hdfs 
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/user/Hadoop/twitter_data/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream 
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0 
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000 

# Describing/Configuring the channel 
TwitterAgent.channels.MemChannel.type = memory 
TwitterAgent.channels.MemChannel.capacity = 10000 
TwitterAgent.channels.MemChannel.transactionCapacity = 100

# Binding the source and sink to the channel 
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel

Unable to figure out whats the problem here

Abhijit
  • 73
  • 11
  • @A. T : I have replied in your previous question, please into into that – Farooque Dec 08 '16 at 10:08
  • this is not the same one , did the correction , – Abhijit Dec 08 '16 at 10:12
  • you are getting java.lang.ClassNotFoundException: org.apache.flume.source.twitter.TwitterSource error... you have to place the flume-source jar in flume lib directory – Farooque Dec 08 '16 at 10:14
  • yes corrected that during previous question only,mentioned in flume.en.sh also getting the same error still – Abhijit Dec 08 '16 at 10:21
  • can you please put that jar in lib directory – Farooque Dec 08 '16 at 10:23
  • i did farooque,i did,still the same error – Abhijit Dec 08 '16 at 10:28
  • can you please check org.apache.flume.source.twitter.TwitterSource class is present in your jar file, because while creating flume agent TwitterSource class is not found, thus causing the error – Farooque Dec 08 '16 at 10:32
  • 1
    Thank you farooque, though i didnt knew how to check for that org.apache.flume.source.twitter.TwitterSource class in jar file,what i did was installed new flume version and downloaded new snapshot jar file.maybe it has that org.apache.flume.source.twitter.TwitterSource class present in it cayse its working – Abhijit Dec 08 '16 at 11:06
  • Congrates......Please accept the previous question's answer...it will help other who might face same problem – Farooque Dec 08 '16 at 11:10
  • @A.T Please copy previous question's answer that worked for this, here. SO Questions are supposed to stand on their own. – josiah Feb 27 '18 at 16:12

1 Answers1

1

Add flume-sources-1.0-SNAPSHOT.jar in flume/lib folder and replace Twitter4j3.0.3.jar with Twitter4j2.2.6.jar files(both core,media and stream files) it may work

Eresh
  • 41
  • 5