1

I am working some more with an RC4 C++ program to be compatible with the command line arguments. I was wondering if somebody can point me in the direction of some decent documentation of how the command line openssl rc4 uses the salts when encrypting vs decrypting, so I can incorporate some compatibility into my program. Thanks to some help from someone on here a few days ago, my program is compatible with the non-salted version, once the EVP_bytestokey function is incorporated. I looked into the EVP_bytestokey function, which openssl uses, and the documentation shows that it can take a salt parameter:

"The salt parameter is used as a salt in the derivation: it should point to an 8 byte buffer or NULL if no salt is used."

I created an array of unsigned char's with the salt that the CLI gave me, and stored them in the array as their ASCII equivalents to pass in (to EVP_bytestokey) as the SALT parameter. Then I compared the file sizes and it shows that the encrypted/salted version of the file is 16 bytes larger than the original file. It appears the CLI version prepends "salted__" to the file, but that only accounts for 8 of the 16 bytes. Does anyone have any idea where the extra 8 bytes would be coming from? As far as I understand, the salt is prepended to the passphrase before passing into the keystream generator for RC4_setkey so I don't understand why the file size should be changed, outside of "salted__".

Also, do you think I am going in the right direction with the SALT array, storing the hex values as ASCII equivalents?

I have the documentation for the C functions being used here, but I can't seem to find anything specific about the mechanics of the CLI version, so I would appreciate any help I can get here.

Chris
  • 95
  • 9
  • 1
    The extra 8 bytes are probably the salt value, according to [enc(1)](https://www.openssl.org/docs/man1.1.1/man1/enc.html) (though that doesn't mention any `salted__` prefix): *When the salt is being used the first eight bytes of the encrypted data are reserved for the salt: it is generated at random when encrypting a file and read from the encrypted file when it is decrypted.* – Shawn Apr 15 '19 at 17:29
  • 1
    [RC4](https://en.m.wikipedia.org/wiki/RC4) has numerous known vulnerabilities and has been deprecated in most use-cases. *Don't* use it in any new code, please. And in old code, please try to get rid of/replace it. – Jesper Juhl Apr 15 '19 at 17:37
  • Yeah, I finally found that on a random page here on SO a few minutes ago. The 16 extra bytes are "salted___" (8 bytes) followed by the actual 8 bytes of the salt, as unsigned char values. Now I am working on getting the format of the salt right in my program. I am having the user input the salt from openssl as the two digit hex values (8 of them), using std::hex on the input to store it into an int variable. From there, I will cast them all to unsigned int's in an array I will then pass into the salt. – Chris Apr 15 '19 at 17:42
  • I got this process to work and output the correct values to the terminal on a little test snippet, so hopefully it correlates well into my actual program. Once I confirm that works, I will snip the first 16 bytes of the salted file, then pass it through the program. Will update on here later today if I am successful. – Chris Apr 15 '19 at 17:42
  • Yes, Jesper, I am aware of the vulnerabilities, and that RC4 is largely being phased out. This is not for any serious implementation. This is just a project I am working on to understand how it works. Once completed, I am integrating my program into and S3FS instance I mounted to my local machine. Won't be used by anybody but me. – Chris Apr 15 '19 at 17:44

1 Answers1

1

So for anyone who is curious, I figured it out. OpenSSL RC4 with salt option prepends 16 bytes to the beginning on the file after encryption. The first 8 bytes are: "salted__" and the second 8 bytes are the actual salt, shown as unsigned char's. Lseek can be used to skip bytes 1-8, then bytes 9-16 can be loaded into an unsigned char array (Has to be size 8 bytes). This array should then be cast to (const unsigned char*) while being passed in as the salt to EVP_bytestokey(), which is the 3rd parameter.

As for a user input option for the salt, I set it up to take in two digit hex inputs into an int variable, which I then pass into the unsigned char salt array.

Chris
  • 95
  • 9