Can't Get Powershell to assign an array over a certain size

Question

So, I'm trying to create a powershell script that "benchmarks" the speed of GPG using various size files. The part in specific that is causing me trouble is this:

1 $temppath="$($env:Temp)\"
...
4 Measure-Command {$bytes= 10000MB
5 [System.Security.Cryptography.RNGCryptoServiceProvider]$rng = New- Object.System.Security.Cryptography.RNGCryptoServiceProvider
6 $rndbytes = New-Object byte[] $bytes
7 [System.Io.File]::WriteAllBytes("$($temppath)\test.txt", $rndbytes)} | Select-Object -OutVariable gentime | Out-Null;

When $bytes >~ 1.5GB (I use 10000MB to test it), I get an error that, If I'm understanding what's going on correctly, corresponds to the fact that the byte array is indexed by a variable of type int32. The specific error is this:

New-Object : Cannot convert argument "0", with value: "10485760000", for "Byte[]" to type "System.Int32": "Cannot convert value "10485760000" to type "System.Int32". Error: "Value was either too large or too small for an Int32.""
At line:6 char:13
+ $rndbytes = New-Object byte[] $bytes
    + CategoryInfo          : InvalidOperation: (:) [New-Object], MethodException
    + FullyQualifiedErrorId : ConstructorInvokedThrowException,Microsoft.PowerShell.Commands.NewObjectCommand

Is there a way to force powershell to index $rndbytes, when the object is created, with an index of type int64 or uint64? If that's not the issue, I'd love to be shown and have what's exactly going wrong explained.

Do note, if you use, for example $bytes=1000MB (or write $bytes=1GB), the code works perfectly. I suspect I'm hitting a wall at $bytes.index > 2^31 -1 or so.

Thanks for the help~!!

No, an array cannot exceed `[int]::MaxValue` in length. Output in batches — Mathias R. Jessen, Feb 08 '21 at 23:56
Dang, that's unfortunate. So just create multiple arrays and process them in batches? — Shinaolord, Feb 08 '21 at 23:56
Create a small buffer array (4KB for example) and then write in chunks of that size until you've reached your desired file size — Mathias R. Jessen, Feb 08 '21 at 23:57
Time to learn how to do that in powershell then. Do you happen to know if `[System.Io.File]::WriteAllBytes` overwrites data, or appends? — Shinaolord, Feb 09 '21 at 00:02

score 1 · Accepted Answer · answered Feb 09 '21 at 00:08

1

As mentioned in the comments, arrays in .NET cannot exceed 2^31-1 items in length, because the index value used for arrays is [int] - the max value of which is 2^31-1.

It's not really a problem, because buffering 2 billion items in memory is terribly inefficient anyways - if you want 10GB of random data, you generate and write it to disk in chunks instead:

# Set file size
$intendedSize = 10GB

# Create output buffer
$buffer = [byte[]]::new(4KB)

# Create RNG
$rng = [System.Security.Cryptography.RNGCryptoServiceProvider]::Create()

# Create temp file, open writable filestrem
$tempFile = [System.IO.Path]::GetTempFileName() |Get-Item
$fileStream = $tempFile.OpenWrite()

do {
    # Fill buffer, write to disk, rinse-repeat
    $rng.GetBytes($buffer)
    $fileStream.Write($buffer, 0, $buffer.Length)
} while($fileStream.Length -lt $intendedSize)

# Dispose if filestream + RNG
$fileStream.Dispose()
$rng.Dispose()

4KB is ususally a good general buffer size on Windows, since 4KB is the default cluster size on NTFS volumes and most modern CPUs have L1 cache sizes divisible by 4KB

answered Feb 09 '21 at 00:08

Mathias R. Jessen

157,619
12
148
206

What is the difference between `$rng = [System.Security.Cryptography.RNGCryptoServiceProvider]::Create()` and `$rng = new [System.Security.Cryptography.RNGCryptoServiceProvider]()` and `[System.Security.Cryptography.RNGCryptoServiceProvider]$rng = New-Object System.Security.Cryptography.RNGCryptoServiceProvider` ? In all honesty, I'm very new to Powershell, I've only really used it for very basic things before this. Also, is Powershell very similar to C#? It seems like it is, and if it is, I might just undertake learning to use C# as a way to supplement my Powershell abilities – Shinaolord Feb 09 '21 at 00:16
@Shinaolord `$rng = new [TypeName]()` is not valid PowerShell. `New-Object TypeName` is essentially the same as `[TypeName]::new()`, but since the `RNGCryptoServiceProvider` exposed a static `Create()` method, the author probably intended for users to use that - so that's what I used :-) PowerShell shares the same underlying base class library as C# (`.NET`) and some syntax element are heavily "borrowed" from C#, but they two distrinct languages. – Mathias R. Jessen Feb 09 '21 at 00:34
I just didn't see the ::Create() method when I looked at microsoft's documentation page for RNGCryptoServiceProvider, so I wasn't sure (:. I really only know Mathematica and less so matlab, and have dabbled in C++ and Python, but by no means am I knowledgeable in those languages. Thanks for the help! – Shinaolord Feb 09 '21 at 00:46

Can't Get Powershell to assign an array over a certain size

1 Answers1