NTE_BAD_DATA in CryptSetKeyParam while setting KP_P in wincrypt

Question

I am having the below code. I am setting a prime for diffie-hellman algorithm using char *. I am getting bad data after i set the prime. Where am i doing wrong? I followed the same example in this link. https://msdn.microsoft.com/en-us/library/aa381969(VS.85).aspx#exchanging_diffie-hellman_keys

What is the correct way to set prime in diffie-hellman using wincrypt?

#define DHKEYSIZE 1024
int fld_sz = 256;
BYTE* g_rgbPrime = new BYTE[DHKEYSIZE/8];   
char * prime = "A1BD60EBD2D43C53FA78D938C1EF8C9AD231F9862FC402739302DEF1B6BEB01E5BE59848A04C48B0069A8FB56143688678F7CC1097B921EA3E13E1EF9B9EB5381BEFDE7BBF614C13827493A1CA31DA76B4083B62C5073451D6B1F06A2F1049C291464AC68CBB2F69474470BBAD374073392696B6447C82BF55F20B2D015EB97B";
string s_prime(prime, fld_sz);
vector<std::string> res;
// split the string two charactes for converting into hex format
for (size_t i = 0; i < fld_sz; i += 2)
    res.push_back(s_prime.substr(i, 2));
for(int i = 0; i < res.size(); i++) {
    BYTE b = static_cast<BYTE>(std::stoi(res[i], 0, 16));
    g_rgbPrime[i] = b;
}
BYTE g_rgbGenerator[128] = 
{
    0x02
};
BOOL fReturn;
HCRYPTPROV hProvParty1 = NULL; 
HCRYPTPROV hProvParty2 = NULL; 
CRYPT_DATA_BLOB P;
CRYPT_DATA_BLOB G;
HCRYPTKEY hPrivateKey1 = NULL;
HCRYPTKEY hPrivateKey2 = NULL;
PBYTE pbKeyBlob1 = NULL;
PBYTE pbKeyBlob2 = NULL;
HCRYPTKEY hSessionKey1 = NULL;
HCRYPTKEY hSessionKey2 = NULL;
PBYTE pbData = NULL;

/************************
Construct data BLOBs for the prime and generator. The P and G 
values, represented by the g_rgbPrime and g_rgbGenerator arrays 
respectively, are shared values that have been agreed to by both 
parties.
************************/
P.cbData = DHKEYSIZE / 8;
P.pbData = (BYTE*)(g_rgbPrime);

G.cbData = DHKEYSIZE / 8;
G.pbData = (BYTE*)(g_rgbGenerator);

/************************
Create the private Diffie-Hellman key for party 1. 
************************/
// Acquire a provider handle for party 1.
fReturn = CryptAcquireContext(
    &hProvParty1, 
    NULL,
    MS_ENH_DSS_DH_PROV,
    PROV_DSS_DH, 
    CRYPT_VERIFYCONTEXT);
if(!fReturn)
{
    goto ErrorExit;
}

// Create an ephemeral private key for party 1.
fReturn = CryptGenKey(
    hProvParty1, 
    CALG_DH_EPHEM, 
    DHKEYSIZE << 16 | CRYPT_EXPORTABLE | CRYPT_PREGEN,
    &hPrivateKey1);
if(!fReturn)
{
    goto ErrorExit;
}

// Set the prime for party 1's private key.
fReturn = CryptSetKeyParam(
    hPrivateKey1,
    KP_P,
    (PBYTE)&P,
    0);
if(!fReturn)
{
    std::cout << GetLastError() << endl;
    goto ErrorExit;
}

// Set the generator for party 1's private key.
fReturn = CryptSetKeyParam(
    hPrivateKey1,
    KP_G,
    (PBYTE)&G,
    0);
if(!fReturn)
{
    std::cout << GetLastError() << endl;
    goto ErrorExit;
}

Thanks in advance.

Update 1: Thanks to @RbMm I was able to set the prime. The problem was with DHKEYSize. However i am getting an error in while setting KP_X. updated the code above to reflect the new code.

Here i converted the string to hex bytes array.

score 1 · Accepted Answer · answered Aug 28 '17 at 22:16

1

size of prime KP_P (and KP_G) and DH key size hard connected. must be cbKey == 8*cbP. look for example Diffie-Hellman Client Code for Creating the Master Key:

as key size if used cbP * 8 where cbP size of prime P. in your link also P.cbData = DHKEYSIZE/8;

also in code instead hard-code size of P (and G) you can get it in runtime:

ULONG dwDataLen;
CryptGetKeyParam(hPrivateKey1, KP_P, 0, &(dwDataLen = 0), 0);
CryptGetKeyParam(hPrivateKey1, KP_G, 0, &(dwDataLen = 0), 0);

and you can sure that dwDataLen == DHKEYSIZE / 8 where DHKEYSIZE is key size.

because you use 512 as key size, the length of data for P and G must be 512/8=64. but you use 256 (for P) and 1 (for G). as result and error.

answered Aug 28 '17 at 22:16

RbMm

31,280
3
35
56

Thanks [RbMm](https://stackoverflow.com/users/6401656/rbmm) . It solved the issue. But i dont have the generator and prime of same length. generator is 1 byte long and prime is 128 byte long. in that case, how can use this one byte generator value? – Prakash N Aug 29 '17 at 14:37
@PrakashN - for generate new **P** and **G** you must not use `CRYPT_PREGEN` flag and next code: `CryptGetKeyParam(hPrivateKey1, KP_P, 0, &(P.cbData = 0), 0) && CryptGetKeyParam(hPrivateKey1, KP_P, P.pbData = (PBYTE)alloca(P.cbData), &P.cbData, 0) && CryptGetKeyParam(hPrivateKey1, KP_G, 0, &(G.cbData = 0), 0) && CryptGetKeyParam(hPrivateKey1, KP_G, G.pbData = (PBYTE)alloca(G.cbData), &G.cbData, 0)` - so generate and get on one side, and on another side use `CRYPT_PREGEN` and this **P** and **G** – RbMm Aug 29 '17 at 17:45
Thanks RbMm. In my case server has already agreed on P and G values. so I need to use the server generated P and G values to generate secret key a. i get the error NTE_BAD_KEY in CryptSetKeyParam of KP_X. Please advice. – Prakash N Aug 29 '17 at 18:06
@PrakashN - *P* and *G* values must be `key_len/8` in size - are your values this kind ? if not it invalid at begin – RbMm Aug 29 '17 at 18:18
I have updated the code to reflect the latest code. when i set to fld_sz to 254 or lower. everything works fine. However for 256 i get NTE_BAD_KEY for KP_X. I want the prime to be of length 256 which is 128 Bytes after converting(1024/8 = 128) – Prakash N Aug 29 '17 at 18:30
the *P* can not be random data. when try set `KP_X` internally called `BCryptSetProperty(..BCRYPT_DH_PARAMETERS..)` and in data blob - data for **P** and **G** . i advice you use more low level [example](https://github.com/Microsoft/Windows-classic-samples/blob/master/Samples/Security/DhOakleyGroup1/cpp/DhOakleyGroup1.cpp) for test with your **P** and **G** - wich status you got at line 229 ? – RbMm Aug 29 '17 at 19:51
P and G are the data i got from server. P is 128 Bytes and G is 1 Byte. Do you think P and G can be of different length? I followed this example https://stackoverflow.com/questions/76581/i-have-p-g-how-do-i-use-the-wincrypt-api-to-generate-a-diffie-hellman-keypai and set the G as 128 Byte array but contains only one Byte. – Prakash N Aug 30 '17 at 15:05
@PrakashN - `Do you think P and G can be of different length? ` i already many time explain this. the length is P and G must be exactly keylen/8. – RbMm Aug 30 '17 at 17:10
Thanks for your time @RbMm. the problem is the different length which is invalidating it. I will make it same length and check it. Thanks. – Prakash N Aug 30 '17 at 18:14
@PrakashN - you must not only formal set length to keylen/8 - but data must be real exactly fit to this len. also data for *P* can not be random bytes.i advice you use more low level example for test with your P and G - https://github.com/Microsoft/Windows-classic-samples/blob/master/Samples/Security/DhOakleyGroup1/cpp/DhOakleyGroup1.cpp the crypto api internal call exactly `BCryptSetProperty` this your P and G data in `BCRYPT_DH_PARAMETER_HEADER` buffer – RbMm Aug 30 '17 at 18:33
Thanks @RbMm. Example you suggested did work for setting the private key. – Prakash N Sep 05 '17 at 20:17

NTE_BAD_DATA in CryptSetKeyParam while setting KP_P in wincrypt

1 Answers1