`Hello Team,
I hope you are good. Actually, I am in need of an assistance regarding Apache flume.
To give you the context, I am newly learning Bigdata and Flume, while loading twitter data into flume I am unable to run fully as I got the below error message in flume log`
Twitter data Conf:
# Naming the components on the current agent.
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
# Describing/Configuring the source
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey = XXXXX
TwitterAgent.sources.Twitter.consumerSecret = XXXXX
TwitterAgent.sources.Twitter.accessToken = XXXXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXXXXX
TwitterAgent.sources.Twitter.keywords = tutorials point,java, bigdata, mapreduce, mahout, hbase, nosql
# Describing/Configuring the sink
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/user/Hadoop/twitter_data/
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1000
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000
TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600
# Describing/Configuring the channel
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 10000
TwitterAgent.channels.MemChannel.transactionCapacity = 1000
# Binding the source and sink to the channel
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel
`In my system, I configured hadoop_home,Java_home & Classpath and I have Twitter Jar files in my lib folder.
[Please see the twitter jar files] (https://i.stack.imgur.com/d3nzj.png)
I got this error from flume log. `
17 Aug 2023 14:44:36,858 INFO [lifecycleSupervisor-1-2] (org.apache.flume.source.twitter.TwitterSource.start:141) - Twitter source Twitter started.
17 Aug 2023 14:44:36,858 INFO [Twitter Stream consumer / [1][initializing]] (twitter4j.SLF4JLogger.info:62) - Establishing connection.
17 Aug 2023 14:44:37,918 INFO [Twitter Stream consumer / [1][Establishing connection]] (twitter4j.SLF4JLogger.info:62) - https://stream.twitter.com/1.1/statuses/sample.json?stall_warnings=true
17 Aug 2023 14:44:37,918 ERROR [Twitter Stream consumer / [1][Establishing connection]] (org.apache.flume.source.twitter.TwitterSource.onException:339) - Exception while streaming tweets
twitter4j.TwitterException: https://stream.twitter.com/1.1/statuses/sample.json?stall_warnings=true
at twitter4j.HttpClientImpl.handleRequest(HttpClientImpl.java:185) ~[twitter4j-core-4.0.7.jar:4.0.7]
at twitter4j.HttpClientBase.request(HttpClientBase.java:57) ~[twitter4j-core-4.0.7.jar:4.0.7]
at twitter4j.HttpClientBase.get(HttpClientBase.java:75) ~[twitter4j-core-4.0.7.jar:4.0.7]
at twitter4j.TwitterStreamImpl.getSampleStream(TwitterStreamImpl.java:201) ~[twitter4j-stream-4.0.7.jar:4.0.7]
at twitter4j.TwitterStreamImpl$4.getStream(TwitterStreamImpl.java:170) ~[twitter4j-stream-4.0.7.jar:4.0.7]
at twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:570) ~[twitter4j-stream-4.0.7.jar:4.0.7]
Caused by: java.io.FileNotFoundException: https://stream.twitter.com/1.1/statuses/sample.json?stall_warnings=true
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_202]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_202]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_202]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1944) ~[?:1.8.0_202]
I appreciate any help you can give. Thanks for your consideration.
