I'm trying to replicate the behaviour of NSS lib's PK11_PrivDecryptPKCS1 function in my code so I can pinpoint why it's failing. It's always returning a SECFailure.
I have seen here that this function creates a parameter and calls pk11_PrivDecryptRaw. Now in pk11_PrivDecryptRaw (same file, line 833) I saw that if can return SECFailure in 3 code blocks:
First if key->keyType != rsaKey. So I debuged key->keyType and saw it has equal to rsaKey.
The next part is this:
session = pk11_GetNewSession(slot,&owner);
(...)
crv = PK11_GETTAB(slot)->C_DecryptInit(session, mech, key->pkcs11ID);
if (crv != CKR_OK) {
(...)
return SECFailure;
}
So I replicated this code in my code in order to debug what's being returned to crv. When I tried to do this I had a lot of trouble finding the .h where some symbols where. It couldn't find the "secmodi.h", "secmodti.h" includes. I found them in the source code for the libnss3-dev.
Now the problem is with the linkage. I'm getting the error
undefined reference for pk11_GetNewSession. I searched the whole /usr/lib dir for this function, with this command:
find . -name \*.so -type f -print -exec readelf -s {} \; 2> /dev/null | grep "\.so\|PK11"
but it didn't find it. I'm thinking it should be and internal function from the library that is not exposed on the .so.
If I try to substitute this call for the code inside this function, I mean
PK11_GETTAB(slot)->C_OpenSession(slot->slotID,CKF_SERIAL_SESSION,slot,pk11_notify,&session)
I get an undefined reference for pk11_notify this time.
How can I debug this ?
