java ByteOrder little endian to big endian not working?

Question

I am trying to decrypt a file in Java which was encrypted using the Microsoft CryptoAPI's CryptEncrypt function. I have read that "the encryption block buffer returned is in little-endian byte order (compared to big-endian for Java and .NET above)."

I am using the ByteBuffer and ByteOrder classes in Java and am pretty sure I am doing it wrong because I am getting the same printout with a System.out.println for beforebytes and afterbytes no matter what I try.

byte [] beforebytes = null;

// code to extract bytes from file here

ByteBuffer bb = ByteBuffer.wrap(beforebytes);

bb.order( ByteOrder.LITTLE_ENDIAN);  // BIG_ENDIAN doesn't work either?

ByteBuffer slice = bb.slice();
// 'slice' now refers to the same data, but is supposed to be BIG ENDIAN

byte[] afterbytes = new byte[bb.capacity()]; 

// transfer bytes from this buffer into the given destination array 
slice.get(afterbytes, 0, afterbytes.length); 

Any help will be greatly appreciated!

Thank you, Bertrand

Answer 1

I resolved this in C! Java now decrypts correctly what was encrypted by the CryptoAPI.

I started out from the CryptoAPI example at: http://blogs.msdn.com/b/alejacma/archive/2008/01/28/how-to-generate-key-pairs-encrypt-and-decrypt-data-with-cryptoapi.aspx

Then just before writing the encrypted text to file, I added a block of code from reference CryptoAPI C++ interop with Java using AES

// reverse bytes of pbData for java 
for (unsigned i = 0; i<dwEncryptedLen / 2; i++)
{
BYTE temp = pbData[i];
pbData[i] = pbData[dwEncryptedLen - i - 1];
pbData[dwEncryptedLen - i - 1] = temp;
}

The reference was for AES but I encrypted in RSA. For decryption I used the bouncycastle provider using algorithm "RSA/NONE/PKCS1Padding". To install the bouncycastle provider on Windows 7, follow: http://sce.uhcl.edu/yang/teaching/JDK_JCE_environment_Configuration.htm and reboot!

Hope this will help someone.

Answer 2

The byte order doesn't matter if you get individual bytes (or a byte array) out of the buffer. It only matters if you are getting for example 16-bit short values or 32-bit integer values out of the buffer; in that case, the bytes from the buffer will be swapped appropriately according to the byte order.

For example:

ByteBuffer buf1 = ByteBuffer.wrap(new byte[]{0x01, 0x02, 0x03, 0x04});
buf1.order(ByteOrder.LITTLE_ENDIAN);

int n1 = buf1.getInt();
System.out.println(n1 == 0x04030201);

ByteBuffer buf2 = ByteBuffer.wrap(new byte[]{0x01, 0x02, 0x03, 0x04});
buf2.order(ByteOrder.BIG_ENDIAN);

int n2 = buf2.getInt();
System.out.println(n2 == 0x01020304);