Decode cryptlib encoded text using Java (without cryptlib)

Question

Hello.

First of all: I'm new to stackOverflow and to the subject I'm talking about...
I am trying to avoid the usage of the cryptlib library for TripleDES encryption in my Java application (now i am using AES - to ensure the downward compatibility I also want to be able to decode the strings which were created with the cryptlib library but without the useage of JNI).
But none of the things I tried by now worked for me.

Configuration:
Algorithm: TripleDES
Mode: CBC
Format: CRYPT_FORMAT_CRYPTLIB
The key has a size of 16 byte (which is inconvenient, but BouncyCastle would support it).
And the encrypted data has a size which is not a multiple of 8 (for exmaple 81 byte).

In my wrapper of the library (also in C) the context is created this way:

cryptCreateContext( &cryptContext, CRYPT_UNUSED, CRYPT_ALGO_3DES);
cryptSetAttribute( cryptContext, CRYPT_CTXINFO_MODE, CRYPT_MODE_CBC );
cryptSetAttributeString( cryptContext, CRYPT_CTXINFO_KEY, key, keyLen);

The envelope is created this way:

cryptCreateEnvelope( envelope, CRYPT_FORMAT_CRYPTLIB );

I think the problem is the format (since it is a "cryptlib native" format) - I can not find any description about it in the net...

Currently my best try was this one:

Security.addProvider(new BouncyCastleProvider());
SecretKeySpec keySpec = new SecretKeySpec(KEY.getBytes(), "DESede");
IvParameterSpec iv = new IvParameterSpec(new byte[8]);
Cipher e_cipher = Cipher.getInstance("DESede/CBC/PKCS7Padding", "BC");
e_cipher.init(Cipher.DECRYPT_MODE, keySpec, iv);

byte[] hexdecoded = Hex.decode(ENCRYPTED.getBytes());
byte [] cipherText = e_cipher.doFinal(hexdecoded);

return new String(cipherText);

I also tried different paddings, but I always end in one of these two Exceptions:

javax.crypto.IllegalBlockSizeException: data not block size aligned
javax.crypto.IllegalBlockSizeException: last block incomplete in decryption

I'm getting a little desperate, so I would be really happy if someone here could help me...

EDIT: Here is the code for the encryption (CryptData.cpp):

bool CCryptData::encryptData(BYTE *key,int keyLen, BYTE *data, int dataLen, int resultMemSize, BYTE *result, int &resultLen)
{
    CRYPT_ENVELOPE cryptEnvelope;
    CRYPT_CONTEXT cryptContext;
    CRYPT_ALGO cryptAlgo = selectCipher( CRYPT_ALGO_3DES );
    int count;

    /* Create the session key context.  We don't check for errors here since
       this code will already have been tested earlier */
    cryptCreateContext( &cryptContext, CRYPT_UNUSED, cryptAlgo );
    cryptSetAttribute( cryptContext, CRYPT_CTXINFO_MODE, CRYPT_MODE_CBC );
    cryptSetAttributeString( cryptContext, CRYPT_CTXINFO_KEY, key, keyLen );

    /* Create the envelope, push in a password and the data, pop the
       enveloped result, and destroy the envelope */
    if( !createEnvelope( &cryptEnvelope ) || \
        !addEnvInfoNumeric( cryptEnvelope, CRYPT_ENVINFO_SESSIONKEY,
                            cryptContext ) )
        return( FALSE );

    cryptSetAttribute( cryptEnvelope, CRYPT_ENVINFO_DATASIZE, dataLen );

    count = pushData( cryptEnvelope, data, dataLen, NULL, 0 );

    if( cryptStatusError( count ) )
        return( FALSE );

    resultLen = popData( cryptEnvelope, result, resultMemSize);

    if( cryptStatusError( count ) )
        return( FALSE );
    if( !destroyEnvelope( cryptEnvelope ) )
        return( FALSE );

    return true;
}

bool CCryptData::checkErrorStatus(int status, CString function)
{
    if( cryptStatusError( status ) )
    {
        m_lastError  = "Error occured in function " + function;
        m_lastError += " with StatusCode: " + status;
        m_bError = true;
        return true;
    }

    return false;
}

int CCryptData::createEnvelope( CRYPT_ENVELOPE *envelope )
{
    int status;

    /* Create the envelope */
    status = cryptCreateEnvelope( envelope, CRYPT_UNUSED, CRYPT_FORMAT_CRYPTLIB );
    if( checkErrorStatus(status, "createEnvelope"))
        return false;

    return( TRUE );
}

int CCryptData::destroyEnvelope( CRYPT_ENVELOPE envelope )
{
    int status;

    /* Destroy the envelope */
    status = cryptDestroyEnvelope( envelope );
    if( checkErrorStatus( status, "destroyEnvelope"))
        return false;

    return( TRUE );

}

int CCryptData::pushData( const CRYPT_ENVELOPE envelope, const BYTE *buffer,
                         const int length, const void *stringEnvInfo,
                         const int numericEnvInfo )
{
    int status, bytesIn;

    /* Push in the data */
    status = cryptPushData( envelope, buffer, length, &bytesIn );
    if( cryptStatusError( status ) )
    {
        printf( "cryptPushData() failed with error code %d, line %d.\n",
                status, __LINE__ );
        return( status );
    }
    if( bytesIn != length )
    {
        printf( "cryptPushData() only copied %d of %d bytes, line %d.\n",
                bytesIn, length, __LINE__ );
        return( SENTINEL );
    }

    /* Flush the data */
    status = cryptPushData( envelope, NULL, 0, NULL );
    if( cryptStatusError( status ) && status != CRYPT_ERROR_COMPLETE )
        {
        printf( "cryptPushData() (flush) failed with error code %d, line "
                "%d.\n", status, __LINE__ );
        return( status );
        }

    return( bytesIn );

}

int CCryptData::popData( CRYPT_ENVELOPE envelope, BYTE *buffer, int bufferSize )
{
    int status, bytesOut;

    status = cryptPopData( envelope, buffer, bufferSize, &bytesOut );
    if( cryptStatusError( status ) )
        {
        printf( "cryptPopData() failed with error code %d, line %d.\n",
                status, __LINE__ );
        return( status );
        }

    return( bytesOut );

}

int CCryptData::addEnvInfoNumeric( const CRYPT_ENVELOPE envelope,
                              const CRYPT_ATTRIBUTE_TYPE type,
                              const int envInfo )
{
    int status;

    status = cryptSetAttribute( envelope, type, envInfo );
    if( checkErrorStatus( status, "addEnvInfoNumeric"))
        return false;

    return( TRUE );

}

CRYPT_ALGO CCryptData::selectCipher( const CRYPT_ALGO algorithm )
{
    if( cryptStatusOK( cryptQueryCapability( algorithm, NULL ) ) )
        return( algorithm );
    return( CRYPT_ALGO_BLOWFISH );
}

EDIT 2: In comparision it is the same implementation as descripted in this mailing list.
But i want to decode it in Java...

EDIT 3: I think the problem is, that the encrypted data is enveloped in cryptlib (as seen in the code).

EDIT 4: I know know that the problem is the enveloping. The cryptlib uses the session key of the created crypt context to envelope the decrypted data in the CRYPT_FORMAT_CRYPTLIB format. The question now is how to decode the envelope before performing the real decryption.
Any suggestions how i could perform the decoding?

score 1 · Accepted Answer · answered May 30 '11 at 08:54

I finally got it.
After debugging the source of cryptlib, I found out that the encrypted text is HEX encoded CMS enveloped content.
I used this online decoder to analyze the ASN.1 sequence (CMS uses ASN.1 notation):
http://www.aggressivesoftware.com/tools/asn1decoder.php

After that I could reproduce the decoding and decryption in Java using BouncyCastle as provider:


    public byte[] decrypt(final byte[] data) throws CryptoException {
        try {
            Security.addProvider(new BouncyCastleProvider());

        EncryptedContentInfo encryptionInfo = parseContentInfo(data);
        AlgorithmIdentifier algoID = encryptionInfo.getContentEncryptionAlgorithm();

        // get the real encrypted data
        byte[] encryptedData = encryptionInfo.getEncryptedContent().getOctets();

        // extract the initialization vector from the algorithm identifier object
        byte[] ivBytes = ((ASN1OctetString) algoID.getParameters()).getOctets();
        // create the key depending on the algorithm
        SecretKeySpec keySpec = new SecretKeySpec(rawKey, algoID.getObjectId().getId());
        // request cipher
        Cipher c = Cipher.getInstance(algoID.getObjectId().getId(), CRYPT_PROVIDER);

        c.init(Cipher.DECRYPT_MODE, keySpec, new IvParameterSpec(ivBytes));
        byte[] decrypted = c.doFinal(encryptedData);

        return decrypted;

        } catch (NoSuchAlgorithmException e) {
            throw new CryptoException(e);
        } catch (NoSuchProviderException e) {
            throw new CryptoException(e);
        } catch (NoSuchPaddingException e) {
            throw new CryptoException(e);
        } catch (InvalidKeyException e) {
            throw new CryptoException(e);
        } catch (InvalidAlgorithmParameterException e) {
            throw new CryptoException(e);
        } catch (IllegalBlockSizeException e) {
            throw new CryptoException(e);
        } catch (BadPaddingException e) {
            throw new CryptoException(e);
        }
    }
}






    private EncryptedContentInfo parseContentInfo(final byte[] encrypted) throws CryptoException {
        try {
            // create a new byte array stream
            ByteArrayInputStream bin = new ByteArrayInputStream(encrypted);
            // create an ASN.1 input stream
            ASN1InputStream ain = new ASN1InputStream(bin);

            // read the whole sequence
            ASN1Sequence mainSequence = (ASN1Sequence) ain.readObject();
            // check if it is an encrypted data
            DERObjectIdentifier mainIdentifier = (DERObjectIdentifier) mainSequence
                    .getObjectAt(ASN1IDENTIFIER_ID);
            if (!mainIdentifier.equals(OID_ENCRYPTED_DATA)) {
                throw new CryptoException("Given data is not encrypted CMS.");
            }
            // parse the encrypted object
            DERTaggedObject encryptedObject = (DERTaggedObject) mainSequence.getObjectAt(ASN1CONTENT_ID);
            // parse the sequence containing the useful informations
            ASN1Sequence encryptedSequence = (ASN1Sequence) encryptedObject.getObject();
            // create the content info object
            EncryptedContentInfo info = EncryptedContentInfo.getInstance(encryptedSequence
                    .getObjectAt(ASN1CONTENT_ID));

            return info;
        } catch (IOException e) {
            // if the main sequence can not be read from the stream an IOException would be thrown
            throw new CryptoException(e);
        } catch (ClassCastException e) {
            // if the parsing fails, a ClassCastException would be thrown
            throw new CryptoException(e);
        } catch (IllegalStateException e) {
            // if the parsing fails, also a IllegalStateException can be thrown
            throw new CryptoException(e);
        }
    }

This way I was able to decode the given text, identify the real decryption algorithm and extract the used initialization vector and the real decrypted data to perform the decryption.

score 0 · Answer 2 · answered May 16 '11 at 22:12

The exceptions appear to lead directly from the ciphertext length you're putting in. By definition CBC outputs a whole number of blocks so it should be giving us a multiple of 8... The question I'd be asking at this point is, what on earth is cryptlib doing? Can you show us the cryptlib code you're using to create the encrypted string?

Decode cryptlib encoded text using Java (without cryptlib)

2 Answers2