0

I have a dashboard web application that currently starts a thread and tails a log file, even time a line is added to the file the tailer picks it up, parses it and then publishes an event around the application, which in turn does things like send it to the client over a web socket or updates a total stored on disk.

This all works fine and so far seems to be handling 500,000 log events a day without batting an eyelid.

But looking at the Spring family maybe there is a better way of doing this in a more 'standardised' fashion that can make it easier to maintain the code and for others to support it.

As I'm new to Spring can someone tell me if this sort of thing is best suited to Spring Integration or Spring XD, or should I take it a stage further and use Spring Integration and Rabbit MQ?

Zac Tolley
  • 2,340
  • 4
  • 19
  • 22

2 Answers2

2

Spring Integration 3.0 now has a tail inbound adapter. It will be released soon, the release candidate was announced last week. XD uses it in its tail source module.

Gary Russell
  • 166,535
  • 14
  • 146
  • 179
  • 1
    ...and Spring XD can be configured to use RabbitMQ as its transport. So if you choose that, the data would be sent via RabbitMQ where the pipe symbol follows a tail source. – mfisher Dec 05 '13 at 22:37
  • If I use the Spring integration tailer isn't that just replacing the tailing library I'm already using, so I still run the same stats code that I do now? Or should I use XD, throw away my code and have some kinda magic XD thingy figuring my stats :) maybe I should just ry and go through a tutorial for the 2 modules and see what sticks. – Zac Tolley Dec 06 '13 at 11:00
0

I'am doing quite the same job with Spring XD. processing 30Go of logs files a day. Enrich them, send them to hadoop and Elasticsearch.

We are really happy with that technology that mixes spring integration, spring batch in a distributed way.

I've noticed that Redis was a real bottelneck and switched to RabbitMq to have a better throughtput

tom
  • 1,647
  • 14
  • 15