4

We can't for the life of us figure this out. We need to make ColdFusion encrypt data which ruby will decrypt. We've tried so many different settings on the ColdFusion side, looked through SO posts, looked through Adobe docs, and cannot make it work. ColdFusion needs to encrypt it so ruby can do this:

aes = OpenSSL::Cipher::Cipher.new('aes-256-cbc').encrypt
aes.key = Digest::MD5.hexdigest("#{password}#{salt}")
aes.iv = Digest::MD5.hexdigest("#{salt}#{password}")[0,16]
encrypted = aes.update(data) + aes.final

ColdFusion pseudo code

key = tobase64(binaryDecode(lcase(hash(password & salt, "md5")), "hex"))
iv = lcase(left(hash(salt & password, "md5"), 16))
encrypt(data, key, "AES/CBC/PKCS5Padding", "Base64", iv)

Tried with/without the tobase64/binaryDecode (saw somebody mention that it would handle conversion back internally or something stupid). lcase is to make it generate MD5s that look like what ruby builds.

What are we doing wrong? Endless bad decrypt on the ruby side

Daniel Huckstep
  • 5,368
  • 10
  • 40
  • 56
  • What are you doing wrong? I don't know. It would help if you detailed the problems you're having (you know, error messages and all that...). – Marnen Laibow-Koser Nov 10 '11 at 18:50
  • 1
    Not to sound sarcastic, but did you compare the basics first? Is the initial input the same: hashed values, iv, key? Also, did you install the unlimited strength policy files? http://kb2.adobe.com/cps/546/e546373d.html – Leigh Nov 10 '11 at 21:06

1 Answers1

1

What are we doing wrong?

You are not being careful with encodings.

You must take encodings into account.

In ColdFusion, you must only use a byte-array as key or IV, and you must only encrypt byte-arrays.

Do not deal with keys, IVs, or cleartexts in any form other than byte-array. Do not deal with them as base64-encoded strings, UTF-16 strings (what Java does by default), or any other form. You must always deal only with byte-arrays, and you must always know the encoding and use the same encoding between ColdFusion and Ruby.

You can get a byte-array from a string using an encoding. I would tend to use the UTF-8 encoding. Look at the CharsetEncode and CharsetDecode functions.

You are also using keys and IVs wrong. Keys may be generated from passwords using an algorithm such as PBKDF2, but only if you don't have a good way of generating with a cryptorandom PRNG and storing them. IVs should be generated with a cryptorandom PRNG, and may be prepended to the ciphertext when you store or transmit it as a convenient method of storing/transmitting the IV too.

yfeldblum
  • 65,165
  • 12
  • 129
  • 169
  • I vastly simplified the process for the question, but once we get it working in a simple case, we can improve the IV/key handling. I originally wanted to use ruby's pkcs5_keyivgen method to handle the key/iv generation from the password, but ColdFusion is really not going to like that (unless you have insight into that as well). – Daniel Huckstep Nov 10 '11 at 21:00
  • Re: *you must only use a byte-array as key or IV* No, with `encrypt(data, ..)` all arguments are strings except `iv` and `iterations` and the `data` argument is interpreted as UTF-8. With `encryptBinary()` the `data` can be a byte array. But I agree using the same charset/encoding is critical. – Leigh Nov 10 '11 at 21:01
  • Leigh, then that's an argument against using the `encrypt` function. What I have done before is to drop down into the `javax.crypto` classes from ColdFusion code and do the encryption on byte-arrays using those classes. It's not hard, you get a lot better control, and it's a lot easier to do encryption correctly and interoperably. – yfeldblum Nov 10 '11 at 22:25
  • Justice - Yeah, I have used both and would agree `javax.crypto` offers more control. But I think the crux of it is understanding the requirements of the tool you are using. If you do not take care with charset/encoding neither option will work as intended ;) – Leigh Nov 11 '11 at 17:19
  • Leight - right. But `javax.crypto` forces you to give it byte-arrays, which means it forces you to deal with encodings, which means it forces you to learn that encodings are a big issue when dealing with portable encryption of text. You have to understand the requirements of the tool you are using. And `javax.crypto` forces you to understand the requirements. – yfeldblum Nov 11 '11 at 18:36
  • Justice - Yeah, I agree it makes you more cognizant of it. But when I first used it, I made similar mistakes there too ;) So all I was suggesting is making an effort to get to know the tool you are using, whether it is `encrypt` or `javax.crypto`. Then you should be able to use either one successfully. – Leigh Nov 11 '11 at 20:38