Event breaking in flume

Question

By default flume considers one line as one event, But I want to do breaking on some other criteria how it can be achieved in flume?

**10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR - An Error has occured for com.marsh.framework.core.exception.MarshException: Record has been modified since last retrieved - Resubmit transaction**

10 Sep 2013 19:43:33,561 [WebContainer : 9] ERROR - handleException():com.marsh.framework.core.exception.MarshException: Record has been modified since last retrieved - Resubmit transaction
 at com.marsh.csa.serviceagreement.ServiceAgreementImpl.updateAgreement(ServiceAgreementImpl.java(Compiled Code))
 at com.marsh.csa.serviceagreementmgmt.CSAManagerImpl.updateCSA(CSAManagerImpl.java(Compiled Code))
 at com.marsh.csa.serviceagreementmgmt.ejb.EJSRemoteStatelessServiceagreementManager_3dcfd156.updateCSA(Unknown Source)
 at com.marsh.csa.serviceagreementmgmt.ejb._ServiceagreementManagerRemote_Stub.updateCSA(_ServiceagreementManagerRemote_Stub.java(Compiled Code))
 at com.marsh.csa.proxy.CSAProxy.updateCSA(CSAProxy.java(Compiled Code))
 at com.marsh.csa.serviceagreement.SaveCSAAction.performAction(SaveCSAAction.java(Compiled Code))
 at com.marsh.csa.serviceagreement.CSAAbstractStrutsAction.execute(CSAAbstractStrutsAction.java(Compiled Code))
 at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(Inlined Compiled Code))
 at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java(Compiled Code))
Caused by: com.marsh.framework.core.exception.MarshException: Record has been modified since last retrieved - Resubmit transaction
 at com.marsh.csa.serviceagreement.ServiceAgreementDAO.updateServiceAgreement(ServiceAgreementDAO.java(Compiled Code))
 at com.marsh.csa.serviceagreement.ServiceAgreementDAO.update(ServiceAgreementDAO.java(Compiled Code))
 at com.marsh.csa.serviceagreement.SAUpdateImpl.updateServiceAgreement(SAUpdateImpl.java(Compiled Code))
 at com.marsh.csa.serviceagreement.SAUpdateImpl.update(SAUpdateImpl.java(Compiled Code))
 ... 26 more
Caused by: com.marsh.framework.core.exception.MarshException: Record has been modified since last retrieved - Resubmit transaction
 at com.marsh.csa.serviceagreement.SaveCSAAction.performAction(SaveCSAAction.java(Compiled Code))
 at com.marsh.csa.serviceagreement.CSAAbstractStrutsAction.execute(CSAAbstractStrutsAction.java(Compiled Code))
 at org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java(Inlined Compiled Code))
 at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java(Compiled Code))
 at org.apache.struts.action.ActionServlet.process(ActionServlet.java(Inlined Compiled Code))
 at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java(Compiled Code))
 at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
 at javax.servlet.http.HttpServlet.service(HttpServlet.java(Compiled Code))
 at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java(Compiled Code))

this is a log file which I am writing to HBase. Line in bold i want as one event and rest as another event . Basically I want to break the events on Date? Is it possible to do in flume ?

Thanks and Regards Chhaya

reference http://stackoverflow.com/questions/16037023/how-to-handle-multiline-log-entries-in-flume — Chhaya Vishwakarma, Dec 30 '13 at 10:18

score 0 · Answer 1 · answered Dec 31 '13 at 16:45

0

@chhay vishwakarma's reference is a good one.

I would also suggest that you look at using a Flume appender for your logging framework. The Flume appender in log4j2 is the best one I'm currently aware of. It's certainly more fully featured than the Log4j appender in Flume itself.

The advantage is that the exception will be written to Flume as a single event and you don't have to attempt to reconstruct it.

answered Dec 31 '13 at 16:45

Sarge

2,367
2
23
36

hi i dont want to use flume appender i have to stream logs from file – Chhaya Vishwakarma Jan 02 '14 at 04:39

Event breaking in flume

1 Answers1