I am working on a small task related to Hadoop and flume. I am trying to fetch the Twitter data using flume but getting the below error
ERROR node.AbstractConfigurationProvider: Sink HDFS has been removed due to an error during configuration java.lang.InstantiationException: Incompatible sink and channel settings defined. sink's batch size is greater than the channels transaction capacity. Sink: HDFS, batch size = 1000, channel MemChannel, transaction capacity = 100
flume_twitter.conf
TwitterAgent.sources = Twitter
TwitterAgent.channels = MemChannel
TwitterAgent.sinks = HDFS
TwitterAgent.sources.Twitter.type = org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sources.Twitter.consumerKey = XXXXXXXXX
TwitterAgent.sources.Twitter.consumerSecret = XXXXXXXXXX
TwitterAgent.sources.Twitter.accessToken = XXXXXXXXXXXXXXX
TwitterAgent.sources.Twitter.accessTokenSecret = XXXXXXXXXXXXX
TwitterAgent.sources.Twitter.keywords = RioCompetition, cpfc, Palace, London, Christmas, New Years
TwitterAgent.sinks.HDFS.type = hdfs
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:9000/TwitterData
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = Text
TwitterAgent.sinks.HDFS.hdfs.batchSize = 100
TwitterAgent.sinks.HDFS.hdfs.rollSize = 0
TwitterAgent.sinks.HDFS.hdfs.rollCount = 1000
# Describing/Configuring the channel
TwitterAgent.channels.MemChannel.type = memory
TwitterAgent.channels.MemChannel.capacity = 100
TwitterAgent.channels.MemChannel.transactionCapacity = 100
# Binding the source and sink to the channel
TwitterAgent.sources.Twitter.channels = MemChannel
TwitterAgent.sinks.HDFS.channel = MemChannel
flume-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
FLUME_CLASSPATH=/home/mauli/Downloads/flume-sources-1.0-SNAPSHOT.jar
below command is used to run the file
flume-ng agent –conf ./conf/ -f ../conf/flume_twitter.conf -Dflume.root.logger=DEBUG,console -n TwitterAgent
also I have tried with the below command
flume-ng agent TwitterAgent -f ../conf/flume_twitter.conf
I have tried so many ways and almost all references of stack overflow
plz help me with this error
Thanks in advance !!!
