Stack Overflow
Stack Overflow

Questions tagged [flume]

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data.

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

1136 questions
-1
votes
1 answer

org.apache.flume.ChannelException: Unable to put batch on required channel: FileChannel ch_file Caused by: java.util.ConcurrentModificationException

there is an error in my flume system, why and how to fix it ?????? org.apache.flume.ChannelException: Unable to put batch on required channel: FileChannel ch_file Caused by: java.util.ConcurrentModificationException! 2016-02-24 17:42:54,715 ERROR…
Tony Kong
  • 1
  • 1
-1
votes
2 answers

Kafka's Consumer can't start

I type the command line as below.but it show it is unable to connect to localhost:2181. I have started the zookeeper. bin/kafka-console-consumer.sh --zookeeper --localhost:2181 --topic pain --from-beginning Exception in thread "main"…
Miedena
  • 123
  • 1
  • 12
-1
votes
1 answer

Pubmed DataSet - Connecting FTP with Flume

To retrieve the datas from Pubmed DataSet [ncbi], I used FireFTP Addon (in firefox) to retrieve the xml, pdf, txt contents. [http://www.ncbi.nlm.nih.gov/pmc/tools/ftp/] I have successfully installed Apache Flume. The main objective is -- I need to…
Soundarya Thiagarajan
  • 574
  • 2
  • 13
  • 31
-1
votes
1 answer

Running flume agent with custom source

I am trying to configure an flume agent with custom source from here, I tried to run flume agent using command flume-ng agent --conf conf --conf-file conf/twitter1.conf --name TwitterAgent But I am unable to start my Flume-agent, it shows error…
riyasbase
  • 3
  • 3
-1
votes
1 answer

How to set flume configuration from terminal in cloudera?

I have uploaded the flume UI of cloudera. We can set the Agent Name and paste the contents of flume.conf there, with the help of this UI. I want to do this task from terminal. My flume.conf file is ready. So how to set agent name and flume…
Pratik Patil
  • 3,662
  • 3
  • 31
  • 31
-1
votes
2 answers

Flume twitter config error

I am trying to extract twitter data using flume. but i am getting the following error 15/04/08 23:16:36 ERROR node.PollingPropertiesFileConfigurationProvider: Unhandled error java.lang.NoSuchMethodError:…
Hussain Shaik
  • 117
  • 1
  • 4
  • 11
-1
votes
2 answers

Flume "OR Regex" Filtering Interceptor

I'm a newbie with Flume and I can't find enough information about regular expressions – that's why I need help with this. I'm trying to make a Regex Filtering Interceptor. I have some events like these:…
Sebastian Loeb Sucre
  • 71
  • 1
  • 1
  • 5
-1
votes
2 answers

Twitter Search API and Hadoop

I need to use Twitter Search API and get twitter data into my HDFS. The Search API uses REST calls and I am trying to understand how I would use that with Apache Flume. I have required keys for authorization to use Twitter's APIs but I just don't…
SaurabJ21
  • 105
  • 3
  • 8
-1
votes
1 answer

Flume Streaming Data from Facebook

Please let me know whether it is possible to get data from Facebook to HDFS using Flume or Not, If yes can you please post the Configuration(Source) for flume. Thanks
Vinodh
  • 1
  • 1
  • 2
-1
votes
1 answer

Apache Flume stopped streaming Twitter data after Twitter API SSL/TLS mandatory requirements.

I have followed cloudera's tutorial (http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/) for an application to ingest twitter data into HDFS using Apache Flume. Twitter has restricted its api connection to SSL/TLS only with…
Hussnain
  • 186
  • 9
-1
votes
2 answers

Log data using flume to required format at sink

I have a requirement in my project. I have to collect log data using flume and that data has to be fed into hive table. Here my requirement to collect files placed in a folder into hdfs which I am doing using spooldir. After this I need to process…
sathish reddy
  • 119
  • 1
  • 9
-1
votes
1 answer

lab exercises on Single node hadoop cluster

I've setup a Single node hadoop cluster node cluster using cloudera's vm and needed to give examples on the below as part of lab exercise, Please help to point to some of the use case of examples that I could use to run the lab exercises on the…
Jaya William
  • 23
  • 1
  • 6
-1
votes
4 answers

XML data via API to Land in Hadoop

We are receiving huge amounts of XML data via API. In-order to handle this large data set, we were planning to do it in Hadoop. Needed your help in understanding how to efficiently bring the data to Hadoop. What are the tools available ? Is there a…
Maverick
  • 484
  • 2
  • 9
  • 20
-1
votes
1 answer

HDFS IO error in Flume

I am trying to load a file from my Windows machine to HDFS using Flume. I am getting the following error: 12:42:02 WARN hdfs.HDFSEventSink: HDFS IO error java.io.IOException: Incomplete HDFS URI, no host:…
user2564690
  • 21
  • 6
-1
votes
1 answer

Not able to get output in hdfs directory using hdfs as sink in flume

I am trying to give normal text file to flume as source and sink is hdfs ,the source ,channel and sink are showing registered and started but nothing is comming in output directory of hdfs.M new to flume can anyone help me through this???????
Antima Pathariya
  • 1
1 2 3
…
75
76