Why does my private key not work to decrypt a key encrypted by the public key?

Question

I'm trying to turn this MSDN page example code into a Powershell script.

The objective is to create files encrypted by a public key for transport into an environment with access to the private key so they can be decrypted.

I'm using New-SelfSignedCertificate to create the certificates because I don't need anyone to trust these certs, and I'm using the full certificate with public and private keys directly out of the Windows Key Store because I'm still just testing the code.

My problem is that when I encrypt, everything seems to work fine, but when decrypting I get the error message:

ERROR: Exception calling "Decrypt" with "2" argument(s): "The data to be decrypted exceeds the maximum for this modulus of 128 bytes."

It looks like New-SelfSignedCertificate is creating public keys of 2048 bits, and even though in the certs mmc snapin I can see that the cert has a private key, I am unable to see any of it's properties, either through UI or via code.

For instance the following code:

$cert = Get-Item 'Cert:\LocalMachine\AddressBook\<ThumbPrint>'
$cert.HasPrivateKey
$cert.PrivateKey

results in

True

and NULL

Here is the code

function ConvertTo-EncryptedFile
{
    [outputType([System.IO.FileInfo])]
    param
    (
        [parameter(Mandatory = $true)]
        [string]$path,
        [string]$client
    )

    $cert = Get-ClientCert -client $client

    if(Test-Path $path)
    {
        $file = Get-Item $path
        $folder = $file.DirectoryName
        $Name = $file.Name

        $destination = Join-Path $folder -ChildPath "$Name.encrypted"

        $serviceProvider = [System.Security.Cryptography.RSACryptoServiceProvider]$cert.PublicKey.Key
        $aesManaged = New-Object System.Security.Cryptography.AesManaged

        $aesManaged.KeySize = 256
        $aesManaged.BlockSize = 128
        $aesManaged.Mode = 'CBC'

        $transform = $aesManaged.CreateEncryptor()

        $keyformatter = New-Object System.Security.Cryptography.RSAPKCS1KeyExchangeformatter $serviceProvider

        [byte[]]$keyEncrypted = $keyformatter.CreateKeyExchange($aesManaged.Key, $aesManaged.GetType())

        [byte[]]$lenK = New-Object byte[] 4
        [byte[]]$lenIV = New-Object byte[] 4

        [int]$lKey = $keyEncrypted.Length
        $lenK = [System.BitConverter]::GetBytes($lKey)
        [int]$lIV = $aesManaged.IV.Length
        $lenIV = [System.BitConverter]::GetBytes($lIV)

        $outFS = New-Object System.IO.FileStream @($destination, [System.IO.FileMode]::Create)

        $outFS.Write($lenK, 0, 4)
        $outFS.Write($lenIV, 0, 4)
        $outFS.Write($keyEncrypted, 0, $lKey)
        $outFS.Write($aesManaged.IV, 0, $lIV)

        $outStreamEncrypted = New-Object System.Security.Cryptography.CryptoStream @($outFS, $transform, [System.Security.Cryptography.CryptoStreamMode]::Write)

        $count = 0
        $offset = 0

        $blockSizeBytes = $aesManaged.BlockSize / 8
        $data = New-Object byte[] $blockSizeBytes
        $bytesRead = 0

        $inFS = New-Object System.IO.FileStream @($path, [System.IO.FileMode]::Open)

        do
        {
            $count = $inFS.Read($data, 0, $blockSizeBytes)
            $offset += $count
            $outStreamEncrypted.Write($data, 0, $count)
            $bytesRead += $blockSizeBytes
        }
        while ($count -gt 0)
        $inFS.Close()
        $outStreamEncrypted.FlushFinalBlock()
        $outStreamEncrypted.Close()
        $outFS.Close()

        $inFS.Dispose()
        $outStreamEncrypted.Dispose()
        $outFS.Dispose()

        Remove-Variable transform
        $aesManaged.Dispose()

        Write-Output (Get-Item $destination)

    }
    else
    {
        throw "File to encrypt not found at path: $path"
    }

}

function ConvertFrom-EncryptedFile
{
    param
    (
        [parameter(Mandatory = $true)]
        [string]$path,
        [string]$client
    )

    $cert = Get-ClientCert -client $client

    if (Test-Path $path)
    {
        $destination = $path.Substring(0, $path.LastIndexOf('.'))
    }
    else
    {
        throw "File to decrypt not found at $path"
    }

    if ($cert.HasPrivateKey)
    {
        $rsaPrivateKey = New-Object System.Security.Cryptography.RSACryptoServiceProvider ($cert.PrivateKey)
    }

    $aesManaged = New-Object System.Security.Cryptography.AesManaged
    $aesManaged.KeySize = 256
    $aesManaged.BlockSize = 128
    $aesManaged.Mode = 'CBC'

    [byte[]]$lenK = New-Object System.Byte[] 4
    [byte[]]$lenIV = New-Object System.Byte[] 4

    [System.IO.FileStream]$inFs = New-Object System.IO.FileStream @($path, [System.IO.FileMode]::Open)

    $inFs.Seek(0, 'Begin')
    $inFs.Seek(0, 'Begin')

    $inFs.Read($lenK, 0, 3)

    $infs.Seek(4, 'Begin')

    $infs.Read($lenIV, 0, 3)

    [int]$lenK = [System.BitConverter]::ToInt32($lenK, 0)
    [int]$lenIV = [System.BitConverter]::ToInt32($lenIV, 0)

    [int]$startC = $lenK + $lenIV + 8
    [int]$lenC = [int]$inFs.Length - $startC

    [byte[]]$keyEncrypted = New-Object System.Byte[] $lenK
    [byte[]]$iv = New-Object System.Byte[] $lenIV

    $inFs.Seek(8, 'Begin')
    $inFs.Read($keyEncrypted, 0, $lenK)

    $inFs.Seek(8 + $lenK, 'Begin')
    $inFs.Read($iv, 0, $lenIV)

    [byte[]]$keyDecrypted = $rsaPrivateKey.Decrypt($keyEncrypted, $false)

}

It stops at decrypting the AES key because I haven't been able to get passed that hurdle yet.

I've tried reducing the AES key size from 256 to 128, but that didn't seem to work, and I don't really want to use a smaller key size anyway, I would rather figure out what's wrong with this code.

Thanks for any help!

Bill

Answer 1

It seems, the problems is that New-SelfSignedCertificate in PowerShell v4 choose provider, which is not appropriate to use with RSACryptoServiceProvider class, and does not have -Provider parameter, that allow specify provider explicitly.

One option to solve this would be to update to PowerShell v5. In PowerShell v5 New-SelfSignedCertificate cmdlet have -Provider parameter, so that you can specify desired provider:

PS> $Cert=New-SelfSignedCertificate -DnsName Test -CertStoreLocation Cert:\CurrentUser\My -Provider 'Microsoft Enhanced RSA and AES Cryptographic Provider'
PS> $Cert.PrivateKey


PublicOnly           : False
CspKeyContainerInfo  : System.Security.Cryptography.CspKeyContainerInfo
KeySize              : 2048
KeyExchangeAlgorithm : RSA-PKCS1-KeyEx
SignatureAlgorithm   : http://www.w3.org/2000/09/xmldsig#rsa-sha1
PersistKeyInCsp      : True
LegalKeySizes        : {System.Security.Cryptography.KeySizes}

To list installed providers you can use following commands:

PS> $Providers=New-Object -ComObject X509Enrollment.CCspInformations
PS> $Providers.AddAvailableCsps()
PS> $Providers|Format-Table Name,Type

Name                                                             Type
----                                                             ----
Microsoft Software Key Storage Provider                             0
Microsoft Passport Key Storage Provider                             0
Microsoft Smart Card Key Storage Provider                           0
Microsoft Base Cryptographic Provider v1.0                          1
Microsoft Base DSS and Diffie-Hellman Cryptographic Provider       13
Microsoft Base DSS Cryptographic Provider                           3
Microsoft Base Smart Card Crypto Provider                           1
Microsoft DH SChannel Cryptographic Provider                       18
Microsoft Enhanced Cryptographic Provider v1.0                      1
Microsoft Enhanced DSS and Diffie-Hellman Cryptographic Provider   13
Microsoft Enhanced RSA and AES Cryptographic Provider              24
Microsoft RSA SChannel Cryptographic Provider                      12
Microsoft Strong Cryptographic Provider                             1

As you need an RSA capable provider, then you need to choose provider with type 1, 12 or 24.