I am trying to fetch twitter events with flume and store them in a tree data structure, that I have created, in order to aggregate these events.
I can start the flume agent in my local machine, using flume node Application, from Java. In my configuration file, as an example, I have specified a file roll sink and the data are written successfully in a file.
Here is the Java program that I start the agent with.
import org.apache.flume.node.Application;
import org.apache.log4j.BasicConfigurator;
import org.apache.log4j.Logger;
public class StreamObserver
{
static Logger logger = Logger.getLogger(StreamObserver.class);
public static void main (String [] args){
BasicConfigurator.configure();
Application.main(args);
//System.out.println(args[0]);
}
}
With program arguments:
agent -n agentName -f flumeConfFile
And here is my config file:
# Name the components on this agent
t_agent.sources = TwitterSrc
t_agent.sinks = Lsink
t_agent.channels = Lchannel
# Describe/configure the source
t_agent.sources.TwitterSrc.type = org.apache.flume.source.twitter.TwitterSource
t_agent.sources.TwitterSrc.consumerKey = **********************
t_agent.sources.TwitterSrc.consumerSecret = *********************
t_agent.sources.TwitterSrc.accessToken = *****************
t_agent.sources.TwitterSrc.accessTokenSecret = *****************
t_agent.sources.TwitterSrc.maxBatchDurationMillis = 200
t_agent.sources.TwitterSrc.keywords = data
# Describe the sink
t_agent.sinks.Lsink.type = file_roll
t_agent.sinks.Lsink.channel = Lchannel
t_agent.sinks.Lsink.sink.directory = destDirectory/
t_agent.sinks.Lsink.fileHeader = true
# Use a channel which buffers events in memory
t_agent.channels.Lchannel.type = memory
t_agent.channels.Lchannel.capacity = 1000
t_agent.channels.Lchannel.transactionCapacity = 100
# Bind the source and sink to the channel
t_agent.sources.TwitterSrc.channels = Lchannel
t_agent.sinks.Lsink.channel = Lchannel
As I have said the agent starts properly and the events are written in a local file.
But I would like to know if there is a way to parse the events from my Java program.
The files created are in a binary format and I would also like to know if there is a way to get them in ascii format so that I can check the results.
Note, I am using IntelliJ IDEA.
