Allocating large arrays; OutOfMemoryException VS OverflowException

Question

Consider the following:

long size = int.MaxValue;
long[] huge = new long[size];     // throws OutOfMemoryException
long[] huge = new long[size + 1]; // throws OverflowException

I know there is a 2GB limit on the size of a single object, which explains the first exception, but why do I get a different exception once the number of elements surpasses 32bits?

(I am using a 64-bit computer if that's important).

EDIT: I can also define and use an indexer that accepts a long with no problems:

internal sealed class MyClass
{
   public object this[long x]
   { 
      get
      {
         Console.WriteLine("{0}", x);
         return null;
      }
   }
}

...

long size = int.MaxValue;
MyClass asdf = new MyClass();
object o = asdf[size * 50]; // outputs 107374182350

Use a generic `List` instead which uses only as much memory as needed. — Tim Schmelter, Jun 08 '12 at 09:55

score 8 · Answer 1 · answered Jun 08 '12 at 09:47

8

C# arrays are indexed by System.Int32. Since size + 1 is beyond Int32.MaxValue, you get an integer overflow.

Use the overload of Array.CreateInstance that takes a long instead, if you really want to use a long as index.

answered Jun 08 '12 at 09:47

user703016

37,307
8
87
112

so, we are permitted to use a `long` variable as an index or size specifier, but only if the value is not greater than 32-bits? that seems odd; shouldn't the compiler require an `Int32`? Why use `long`s as indexes at all if 32-bits is the index limit? – Dave Cousineau Jun 08 '12 at 09:56
1

Through array subscripting (indexers: `[index]`) you can only use `int`. If you want to use a `long` you have to use the `GetValue` and `SetValue`methods. – user703016 Jun 08 '12 at 09:57
3

It's worth pointing out that `Array.CreateInstance` doesn't solve the problem. You still cannot create too large an array. A related question is asked [here](http://stackoverflow.com/questions/10945404/why-is-long-being-allowed-as-array-length-in-c/10945507#10945507), as of .NET 4.5, larger arrays are supported on 64-bit systems. All integral types can be used as array initializers at compile time. – Adam Houldsworth Jun 08 '12 at 09:57
@Cicada I seem to be able to use a `long` indexer. – Dave Cousineau Jun 08 '12 at 10:01
@Sahuagin The array size limitation is a CLR shortcoming, not a language shortcoming. Language support sort of exists, it's the runtime that usually gives you grief. This is changing as of .NET 4.5. – Adam Houldsworth Jun 08 '12 at 10:04
@Cicada Yes, I definitely can define and use a `long` indexer, and I can even pass it a value equal to or greater than `int.MaxValue + 1` and use it. – Dave Cousineau Jun 08 '12 at 10:07
@Sahuagin Can you please post the code for that? I cannot achieve that using a console app targeting 64-bit, I can't even create the array - runtime errors. You can use all integral types as indexers and size initializers, it's the actual value inside them that is checked at runtime. – Adam Houldsworth Jun 08 '12 at 10:11
@AdamHouldsworth I can't instantiate an array that big, but I can define and use an indexer property using a value that big. I have posted an example. – Dave Cousineau Jun 08 '12 at 10:18
@Sahuagin A custom indexer can take anything as a parameter. We are talking about arrays. They are not comparable. Either way, a long can be supplied to an array, but as yet it would be pointless. – Adam Houldsworth Jun 08 '12 at 10:20
@AdamHouldsworth Ok sorry for the confusion, but that is what 'indexers' are; I have not seem them called 'custom indexers'; they are simply 'indexers'. It seems like this all boils down to differences between what C# and CLR specifically allow. – Dave Cousineau Jun 08 '12 at 10:24
2

@Sahuagin They are the same, the difference is you coded the indexer in your example and the BCL team coded the index on the array. The question was "can you index arrays using `long`", not can indexers use `long`. – Adam Houldsworth Jun 08 '12 at 10:27
@AdamHouldsworth well, you can index arrays using `long`, you'll just get an exception if the actual value exceeds 32bits. So they must cast the `long` down to an `int` somewhere I guess. – Dave Cousineau Jun 08 '12 at 10:37
1

@Sahuagin Yes, as I've stated a few times, you can use any integral type as indexers and size initializers so long as the value is not too large (or negative). – Adam Houldsworth Jun 08 '12 at 10:38

score 2 · Accepted Answer · answered Jun 20 '12 at 02:01

So from what I've gathered, something like the following is happening here:

Indexers in general could use any type of parameter.
The built-in array indexers can accept any integral type...
But the underlying implementation of the built-in array indexers requires a value that is <= Int32.MaxValue and will throw an overflow exception for a value that exceeds Int32.MaxValue.

While the latter point feels like some kind of weird contradiction (accepting types larger than Int32, but throwing an exception if you happen to actually use any of those extra bits), it is apparently a side-effect of the fact that some of this is half-implemented for the future implementation of arrays which will be allowed to have more than Int32.MaxValue elements.

Allocating large arrays; OutOfMemoryException VS OverflowException

2 Answers2

Linked