22

I have an open source library which is designed to support very large collections efficiently. I have tested it on my PC with 24 GB but would like to test it on much a larger system, up to 1 TB. Most hosting solutions don't support this sort of memory sizes and I only need access for brief periods of time.

What I have tested is a collection of 500 million objects with 12 fields the full GC time is below 0.11 seconds. I have another test were it stores 128 billion elements with just one bit.

The library/test is small, so I don't have require much other than a lot of main memory.

Do you have any suggestions on how I could do this testing without buying my own server with 96 or 192 GB?

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • 10
    I've got to ask: If you're having trouble finding a suitable environment to perform this kind of testing in, doesn't that say something about the validity of the tests themselves? Not to rain on your parade, but why bother testing with these insane amount of objects, if there's little chance that any of your users will every utilize your library in such an extreme way? If you do have a (potential) user, there is a good chance that they'll be able to provide you with a suitable test environment (as that should somewhat mimic their production environment). – Guus Nov 16 '11 at 14:53

4 Answers4

8

EC2 has high-memory instances of up to 68.4 GB each, and they charge by the hour. Granted, that is not 100GB of memory, but, if you stack a few of them up together....

C. K. Young
  • 219,335
  • 46
  • 382
  • 435
  • 1
    The problem is running it in one JVM. ;) It may be the best option. I was hoping there might be others. – Peter Lawrey Aug 09 '11 at 19:36
  • Well, use a _real_ JVM that lets you cluster around several machines! ;-P – C. K. Young Aug 09 '11 at 20:54
  • 1
    The purpose is to test performance, and using multiple machine which change the behaviour. It appears I can buy a machine with 128 GB for about £3K. I would rather not, just to test software. – Peter Lawrey Aug 10 '11 at 06:33
  • Another quote is £1.8K for a 96 GB machine. – Peter Lawrey Aug 10 '11 at 07:06
  • Another quote for £1.3K for a 72 GB machine. – Peter Lawrey Aug 10 '11 at 11:10
  • @Peter: I think if you're going to get a 72GB machine, that's awfully close to the 68.4GB instances, and you will probably find EC2 more cost-effective unless you're doing your testing 24/7 over an extended period of time. :-) – C. K. Young Aug 10 '11 at 18:33
  • I will look at using a EC2. While its not as large as I would like, it is perhaps as large as most systems will get for now. – Peter Lawrey Aug 11 '11 at 08:34
  • @PeterLawrey Extremely large memory sizes do have problems with your default Hotspot VM, i.e. there's a reason [azul system](http://www.azulsystems.com/) exists (well and to host Cliff's blog which is all I'd need anyhow ;) ). As I understand the GC times just get plain horrible for large heaps and consistency/latency in general suffer quite a bit. So I don't know how often your software would encounter such large single JVMs in reality. – Voo Nov 09 '11 at 20:44
  • 1
    @Voo, I agree the heap size is inherently limited. This is why the library uses direct memory and memory mapped files. These incur only a trivial GC hit. A JVM can use 32 GB of memory with a heap that's smaller than 16 MB. – Peter Lawrey Nov 10 '11 at 07:44
8

Contact Contegix - they may be able to help you for free since you have an open source library you need to test. I reached out to them recently for the same need and they followed up within a day via email and called the day after and were very open to the possibility of the need to use large amounts of memory spanned across several servers for scale out testing.

Their Advocates for Innovation page describes who they help already. Just fill our their contact form at http://www.contegix.com/contact/ and they should get back to you soon.

Jim Bethancourt
  • 1,061
  • 11
  • 16
1

Perhaps you could use a "virtualized" JVM like Zing -- it has a maximum heap size of 512 GB. Maybe if you contact them they will let you perform this test for free -- it would be a great showcase for their garbage collector.

Artur Nowak
  • 5,254
  • 3
  • 22
  • 32
0

You should take a look at Amazon EC2 or Google app engine.

Mathias Bak
  • 4,687
  • 4
  • 32
  • 42