Getting SEC_ERROR_BAD_PASSWORD error on using libnss API PK11_Authenticate()

Question

We have a use case to list certificates from USB dongles (ePass2003Auto and such types) which will be used for signing PDFs.

Env: Ubuntu 20.04

Libs: libnss* APIs and others like libssl libcrypto libnss3 libnssckbi libnssdbm3 libnssutil3 libnspr

Steps done so that LibreOffice detects/lists the certificate:

The lib/driver to interact with USB dongle is available on the vendor's site. For example in our case --> https://www.hypersecu.com/updates-india . This needs to be installed as per the mentioned instructions. We will have libcastle_v2.so.1.0.0 installed in /usr/lib
Create a nssdb path like /home/<user>/.pki/nssdb
Using modutil (a libnss util) link the created nssdb path, usb slot/token and lib with command:

modutil -dbdir -add "<token name>" -libfile /usr/lib/libcastle_v2.so.1.0.0

To confirm whether it has been added use the command:

modutil -dbdir sql:<nssdb path> - list

LibreOffice (dev - 7.3.6.2):

Tools --> Options --> Security --> Certificate
Set certificate path pointing to the nssdb path
Restart the application
File --> Digital Signatures
Enter USB token password
Certificate will be listed

What we tried:

PK11_SetPasswordFunc( GetPasswordFunction ) ; // GetPasswordFunction() returns the password of USB token

if (NSS_InitReadWrite("/home/<user>/.pki/nssdb") == SECSuccess)  // Initialize nssdb
{
    printf("\n----------->NSS_Init successful");

    cert_handle = CERT_GetDefaultCertDB();

    if (cert_handle == NULL)
    {

        error = PR_GetError();
        printf("\n----------->Unable to get cert_handle:%s (%d)", PR_ErrorToName(error), (int)error);
    }
    else
    {
        printf("\n----------->Got cert_handle");

        PK11SlotList * slotList = PK11_GetAllTokens( CKM_INVALID_MECHANISM, PR_FALSE, PR_FALSE, NULL ) ;

        if(slotList == NULL) {

            error = PR_GetError();
            printf("\n----------->PK11_GetAllTokens failed !:%s (%d)", PR_ErrorToName(error), (int)error);

        } else {

            PK11SlotInfo * usb_token = NULL;

            for (PK11SlotListElement* slotEle = slotList->head ; slotEle != NULL; slotEle = slotEle->next)
            {
                PK11SlotInfo * pSlot = slotEle->slot ;

                if(pSlot != NULL)
                {
                    printf("\n----------->SlotName(%s) TokenName(%s)", PK11_GetSlotName(pSlot), PK11_GetTokenName(pSlot));

                    if(PK11_IsHW(pSlot) && PK11_IsRemovable(pSlot)){ // select the USB-token in the slots list

                        usb_token = pSlot;
                        break;
                    }
                } else {
                    printf("\n----------->pSlot is empty");
                }
            }// end of for

            if(usb_token != NULL){

                printf("\n----------->Found USB-TOKEN SlotName(%s) TokenName(%s)", PK11_GetSlotName(usb_token), PK11_GetTokenName(usb_token));

                if (PK11_NeedLogin(usb_token)){

                    SECStatus nRet = PK11_Authenticate(usb_token, PR_TRUE, NULL);

                    if(nRet != SECSuccess){
                        error = PR_GetError();
                        printf("\n----------->PK11_Authenticate failed !:%s (%d)", PR_ErrorToName(error), (int)error);
                        printf("\n----------->PORT_GetError() !:(%d)", PORT_GetError());

                        if(PORT_GetError() != SEC_ERROR_IO) {
                            printf("\n----------->NoPassword Exception");
                        } else {
                            printf("\n----------->Some other Exception");
                        }
                    }else {
                        printf("\n----------->PK11_Authenticate successful !");
                    }
                }
            }

            PK11_FreeSlotList(slotList);
        }
    }

    PK11_SetPasswordFunc( NULL ) ;
    PK11_LogoutAll();
    NSS_Shutdown();
}
else
{

    error = PR_GetError();
    printf("\n----------->NSS_Init failed:%s (%d)", PR_ErrorToName(error), (int)error);
}

We get the following error even though the returned password in GetPasswordFunction() is correct: PK11_Authenticate failed !:SEC_ERROR_BAD_PASSWORD (-8177)

Any help is appreciated! Thanks in advance!

score 0 · Answer 1 · answered Nov 11 '22 at 05:41

Solved this!

While debugging PK11_Authenticate() and related functions in the flow, it was identified that the password returned from password callback function (as per my sample code --> GetPasswordFunction()) was NULL. This was odd as I had already debugged that function and made sure that the password was properly returned.

On checking other functions in libnss, I came to know about PORT_Alloc() which is a memory allocator (secport.h) similar to malloc(). I had used normal C-style string memory allocation. On trying the allocation with PORT_Alloc(), it worked!

Getting SEC_ERROR_BAD_PASSWORD error on using libnss API PK11_Authenticate()

1 Answers1