Looking for simple persistent message buffer in Java

Question

I am looking for a simple persistent buffer as temporary storage for JSON messages in a Java application. Memory usage should be relatively constant and not depend on the number of messages in the buffer. It would nice to be able to replay messages from a point in the past. Deletion of old messages should be efficient. Needs to be able to handle 1m messages/h.

Currently my application uses a local RabbitMQ broker which shovels messages to a remote RabbitMQ broker. When the remote broker is down or not accepting messages the local RabbitMQ broker's memory usage rises with the queue length and eventually it stops accepting messages. I want to swap this out for a local disk based buffer and a thread copying messages to the remote RabbitMQ broker.

Anyone have any ideas? I have looked at Kafka but it seems like overkill for my use-case. MongoDB is a possibility but I am worried about its memory usage.

redis is blazing fast but needs so much memory. check this out. http://nosql.mypopescu.com/post/1010844204/redis-memory-usage — Erhan A, Aug 13 '12 at 14:19
You might consider something like https://github.com/peter-lawrey/Java-Chronicle Its designed to support over 10M messages per second but you have to rotate the files to delete them. — Peter Lawrey, Aug 13 '12 at 14:23
The app already uses Redis for other things but it uses too much memory for this — David Tinker, Aug 13 '12 at 14:51
Java-Chronicle looks like it might do the job .. will check it out tm! Thanks — David Tinker, Aug 13 '12 at 15:06

Erhan A · Accepted Answer · 2012-08-13T14:17:42.350

Memory usage is always an issue in any system.I am using MongoDB for production and when I compare with similar solutions (CouchDB,CouchBase,redis.io), MongoDB is really good in memory management and easiness of implementation. But I should admit , I never had a chance to test Riak more in detail.

I am storing 5.000.000 user records with 4 index fields and all user session behind a rest/web service api which uses a messaging service behind.

My messaging service uses another db instance on the same server. My user records have at least 20 fields and session records have just 5 fields. My ubuntu servers never used more than 10 GB rams even with heavy loading processes.

Hope this helps to figure out.

ps: all depend on data model and how you implement your infrastructure.

Regards,

EDIT:

I think this is a good slideshow about using MongoDB for messaging.

and a nice article about MongoDB and messaging.

You can use the test code and see the results are ok for your solution. Please don't forget to share your results if you test.

I ended up writing a persistent message queue server with the replay stuff we needed. Checkout http//qdb.io/ — David Tinker, Jul 31 '13 at 21:02

Looking for simple persistent message buffer in Java

1 Answers1