0

Am working on custom solution where I need to store huge data to file/db at the speed of 60K records per second. This data is result of incoming continuous stream.

  1. leveldb - it can NOT be accessed simultaneously from multiple java processes

  2. berkleydb/kyoto cabinet : prohibitive commercial license

  3. sqlite : Tried sqlite4java but it doesnt support bulk operations and is not performing at the required speed. Also tried jdbc wrapper(http://www.zentus.com/sqlitejdbc/) this also doesnt perform at the required speed.

Can someone please suggest me solution which will just allow me to dump data to db/file and can be accessed by multiple processes(+threads)?

Ketan Khairnar
  • 1,620
  • 3
  • 12
  • 21

5 Answers5

1

Can someone please suggest me solution which will just allow me to dump data to db/file and can be accessed by multiple processes(+threads)?

If you want access from multiple processes, multiple threads and/or both, then it's very difficult to reach your goal of 60k records per second simply because disk I/O is not very efficient when you're using multiple threads/processes. Any solution that you have would need to reduce to the same basic model: only one thread can write to LevelDB.

If you want cross-process access to LevelDB, then you can write a very simple communication layer between your processes using sockets or pipes. You would dedicate a single process to accept multiple socket or pipe connections from all the client processes and you would accept streaming data. You then write your data to LevelDB from that process- let's call that the data process.

The data process will have multiple threads that supply data, but only one reading/writing thread that reads/writes data into LevelDB, so it might also be a good idea to use a BlockingQueue which the client threads would enqueue records to and the data thread will read records from (blocking if there is no data).

Kiril
  • 39,672
  • 31
  • 167
  • 226
0

Try RocksDB by Facebook. It can be accessed from java and from multiple threads. It has better performance than LevelDB.

Andrejs
  • 26,885
  • 12
  • 107
  • 96
0

You could try MySQL InnoDB. I can't say if its fast enough for you, but multiple processes can access different tuples of data (locking is per row).

donatello
  • 5,727
  • 6
  • 32
  • 56
0

Try Redis, is very fast, you have the possibility to save your data on ram and later to disk, good solution for you problem i think.

Here the clients lib for java.

byterussian
  • 3,539
  • 6
  • 28
  • 36
0

Try sqlite-jdbc library http://code.google.com/p/sqlite-jdbc/ It is a JDBC wrapper on native dll/so implementations.

Napte
  • 167
  • 6