-1

We are working on a solution which crunches log files generated by systems and does various analysis operations on these logs to come up with different views which can help to triage the issues. For e.g. building a sequence of error messages which are repeating across the logs.

Currently we are loading the logdata in java collections and doing all operations by iterating/searching through these collections which is affecting the performance. We are thinking to instead load the data in a database and fire queries on the data to get optimized search results. And for the same we are thinking on using an in-memory db which will give a better performance than a persistent store as disk reads/writes will be minimized.

The amount of data to be analyzed at a time may go up to few GBs (2-4 GBs) and hence may exceed the RAM size on the machine.

Question: What options can be considered for such an In-Memory db? Is GridGain a good option for the same?

Most of our solutions shall be deployed on a single node and hence distributed capabilities are not the priority. What other in-memory db's can be recommended for this purpose

user398039
  • 143
  • 1
  • 12
  • i have no experience with GridGain. We use http://www.h2database.com/html/main.html which can create in-memory databases: http://www.h2database.com/html/features.html#in_memory_databases – Moh-Aw Jul 10 '14 at 08:47
  • How about Redis? You'll need persistence anyways, since you can't rely on it all fitting it in the memory. If your dataset will be growing, you should look for a distributed solution since it'll scale easier. – Kayaman Jul 10 '14 at 08:53
  • Thanks Moh-Aw. Do you feel an In-Memory db is the right solution for this use case? – user398039 Jul 10 '14 at 08:53
  • Thanks Kanyaman. Yes I may require scaling abilities in the future. But my only concern is will an In-memory db a right choice if I have to deploy it on a single node also? And I dont need data persisntence of the output. The data persistence required to load more data in less RAM will I suppose be handled by in-memory Db by swapping files onto disk as required? – user398039 Jul 10 '14 at 08:55
  • 1
    While I agree that you should make the best use of the hardware you got, perhaps it is time to get more memory. I bought my 8 year old son 8 GB for his machine 1.5 years ago, just because 4 GB wouldn't have made much difference. If you are trying to do something professional with a database, perhaps 32 GB should be your minimum. You can buy 32 GB for $400 and 16 GB for half that. – Peter Lawrey Jul 10 '14 at 09:51

1 Answers1

1

You could try column store in-memory databases. They usually can achieve better compression ratio than row store db, and are designed for analytical tasks. Examples are MonetDB (open source), Vertica, InfiniDB and so on.

lgylym
  • 206
  • 1
  • 8