1

We have some very large data files (5 gig to 1TB) where we need quick read/write access. Since we have a fixed record size it seems like some form of ISAM would be the way to go. But would be happy to hear other suggestions.

Ideally the solution would have an Apache or LGPL style license but we will pay if we have to.

Must haves:

Scalable - over at least 1 TB files
Stable - either doesn't corrupt data or has fast recovery process
Runs well on X64 Windows

Nice to have:

Can participate in 2 phase commits
Intrinsic compression facilities
Portable to *nix platforms
C# API or Java API

Thanks,

Terence

Terence
  • 337
  • 4
  • 12

2 Answers2

3

You can also use the ESENT database engine which is built into Windows. As far as your requirements go:

  • Scalable: the maximum database size is 16TB. Multi-TB datbases have been used in production.
  • Stable: crash recovery with write-ahead logging.
  • X64 Windows: ESENT is part of Windows, so it is present on your 64-bit machine.

Nice to have:

  • 2 phase commits: No.
  • Compression: No.
  • Portable to *nix: No.
  • C# API or Java API: Not really (there is a C# interop layer on Codeplex but it isn't complete).

The documentation is here: http://msdn.microsoft.com/en-us/library/ms684493(VS.85).aspx

You can get the header file and lib by downloading the Windows SDK.

hfcs101
  • 316
  • 1
  • 3
  • Sorry for the tardy thank you and up vote. Its not a good fit for this project - but I may be able to use the information elsewhere. Embarrassing that I didn't know about ESENT given how long I have been coding on Windows. – Terence Jun 05 '09 at 14:46
  • Note that in Windows 7 and Server 2008 R2 ESENT now supports compression. – Laurion Burchall Mar 31 '10 at 05:52
  • And with more modern windows versions the page size of 16 and 32 kb is also available ... should allow up to 64 Terabyte nowadays. Really cool for a hidden database within windows os. – nabuchodonossor Feb 04 '20 at 09:50
1

Give Berkeley DB a try. Opinions vary, but it's scalable, stable (if you use all necessary layers) and AFAIK runs well on x64 windows. Also portable to *nix and has C and Java API. Don't know about C# API.

tzot
  • 92,761
  • 29
  • 141
  • 204
  • Thanks - it looks promising and its use by Subversion bodes well for the ability to scale. Someone just released a C# wrapper - http://www.oreillynet.com/xml/blog/2005/12/net_c_wrapper_for_berkely_db_n.html – Terence Sep 27 '08 at 17:28