Python 3.2 - Symétric AES Crypto in CBC mode - advices requested

Question

My goal is to setup simple encryption with python 3.x, so I searched the web this weekend to get informations about RSA / AES and so on ... Actually, things that may look like a possibility to encrypt text data for transmission in a reasonable secure way .. without paranoia either, I'm not an expert just want to make sure that the stuff is pretty hard to read without the key !

Honestly, I do not know much about cryptography. After several hours of searching and collecting information and source code, my attempts failed because of invalid length problems or other conversion errors due to the examples provided in Python 2.7 . I found very few examples in python 3 and encryption methods used seemed to be not really appropriate or serious me.

I was finally was able to run the following code that accepts ISO 8859-1 coded characters. I actually encapsulates everything in UTF-8 encoding to avoid language issues .. I hope so ....

I would like to know if I'm on the right way of design and especially if data security is acceptable, again I'm not looking for the Great security solution, just want to protect my own personnal data and not to protect a military defense secret Lol !

Feel free to forward me your comments or suggestions and especially things I could have missed !

Thanks very much per advance.

Emmanuel (France)

Note: next step I'll try to send an RSA encrypted AES password to the recipient along with the text stream. As the AES password is different for each message the client needs to automatically translate it to be able to decode the cipher message. The AES password will be transmitted in RSA asymmetric encryption with the strongest possible key without performance breakdown. The aim is to transmit simple messages ( w/o base64 encoding) or large volumes of data in a reasonable timeframe.

@+ see you.

To execute the code bellow, you should have PyCrypto (python 3.2) installed

import os, base64, hashlib
from Crypto.Cipher import AES

class Aes(object):

# Crypte / décrypte un texte donné en AES mode CBC. Accepte l'encodage base64.
# Encrypts input text string & decrypts bytes encoded string with or without base64 encoding
# Author: emmanuel.brunet@live.fr - 12/2013


SALT_LENGTH = 64
DERIVATION_ROUNDS=10000
BLOCK_SIZE = 16
KEY_SIZE = 256
MODE = AES.MODE_CBC

def encrypt(self, source, aes_key, outfile=None, base64_encode=False):
    '''
    Crypte l'entrée source en AES mode CBC avec sortie encodage base64 / fichier facultative

    @param str source: text to encode or text file path
    @param bytes aes_key: password
    @parm str outfile: disk file to write encoded text to. defaults to None
    @param bool base64_encode: returns base64 encoded string if True (for emails) or bytes if False

    @return bytes ciphertext: the bytes encoded string.
    '''


    '''
    ----------------------------
    Inputs management
    ----------------------------
    '''
    if os.path.exists(source):

        fp = open(source, 'rb')
        input_text = fp.read()
        fp.close()

    else:

        input_text = bytes(source, 'UTF-8')

    if input_text == b'':
        print('No data to encrypt')
        return

    padding_len = 16 - (len(input_text) % 16)         
    padded_text = str(input_text, 'UTF-8') + chr(padding_len) * padding_len

    '''
    ---------------------------------------------------------
    Computes the derived key (derived_key). 
    ---------------------------------------------------------
    Elle permet d'utiliser la clé initiale (aes_key) plusieurs 
    fois, une pour chaque bloc à encrypter.
    ---------------------------------------------------------
    '''

    salt = os.urandom(self.SALT_LENGTH)

    derived_key = bytes(aes_key, 'UTF-8')     

    for unused in range(0,self.DERIVATION_ROUNDS):

        derived_key = hashlib.sha256(derived_key + salt).digest()

    derived_key = derived_key[:self.KEY_SIZE]

    '''
    ----------------
    Encrypt
    ----------------
    '''      
    # The initialization vector should be random
    iv = os.urandom(self.BLOCK_SIZE)

    cipherSpec = AES.new(derived_key, self.MODE, iv)
    cipher_text = cipherSpec.encrypt(padded_text)
    cipher_text = cipher_text + iv + salt

    '''
    -------------------------
    Output management
    -------------------------
    '''
    if outfile is None:
        '''
        Returns cipher in base64 encoding. Useful for email management for instance
        '''
        if base64_encode:
            return(base64.b64encode(cipher_text))
        else:
            return(cipher_text)

    else:
        '''
        Writes result to disk
        '''

        fp = open(outfile, 'w')

        if base64_encode:
            fp.write(base64.b64encode(cipher_text))
        else:
            fp.write(cipher_text)

        fp.close()

        print('Cipher text saved in', outfile)


def decrypt(self, source, aes_key, outfile=None, base64_encode=False):
    '''
    Decrypts encoded string or data file

    @param bytes or str source: encrypted bytes string to decode or file path        
    @param bytes aes_key: password
    @parm str outfile: disk file to write encoded text to. defaults to None        
    @param bool base64_encode: cipher text is given base64 encoded (for mails content for examples)

    @returns str secret_text: the decoding text string or None if invalid key given
    '''

    '''
    ---------------------------
    Input management
    ---------------------------
    '''

    if type(source) == str and os.path.exists(source):

        fp = open(source, 'rb')
        ciphertext = fp.read()
        fp.close()

    elif type(source) == bytes:
        ciphertext = source

    else:
        print('Invalid data source')
        return

    if base64_encode:
        encoded_text = base64.b64decode(ciphertext)
    else:
        # decodedCiphertext = ciphertext.decode("hex")
        encoded_text = ciphertext

    '''
    -------------------------
    Computes derived key
    -------------------------
    '''

    iv_start = len(encoded_text) - self.BLOCK_SIZE - self.SALT_LENGTH
    salt_start = len(encoded_text) - self.SALT_LENGTH
    data, iv, salt = encoded_text[:iv_start], encoded_text[iv_start:salt_start], encoded_text[salt_start:]

    derived_key = bytes(aes_key, 'utf-8')

    for unused in range(0, self.DERIVATION_ROUNDS):
        derived_key = hashlib.sha256(derived_key + salt).digest()

    derived_key = derived_key[:self.KEY_SIZE]


    '''
    -------------------------
    Decrypt
    -------------------------
    '''
    Cipher = AES.new(derived_key, self.MODE, iv)
    padded_text = Cipher.decrypt(data)

    padding_length = padded_text[-1]
    secret_text = padded_text[:-padding_length]

    '''
    Si le flux n'est pas décodé (mot de passe invalide),  la conversion UTF-8 plante ou au mieux on obtient un texte illisible
    '''
    try:
        secret_text = str(secret_text, 'utf-8')
    except:
        return

    if outfile is None:

        return(secret_text)

    else:
        '''
        Writes result to disk
        '''
        fp = open(outfile, 'w')
        fp.write(secret_text)
        fp.close()

final stuff

I've made the following changes:

uses PBKDF2 as KDF with HMAC-sha512
Fixed the constant issue
mandatory packages are now : PyCypto & pbkdf2-1.3

I've tried many time to insert the new code block ... but it doesn't work. Very strange behaviour of the text editor.

Thanks Owlstead, as mentioned previously by Faust I've changed the derivation key computing by PBKDF2. — Emmanuel BRUNET, Dec 10 '13 at 19:37
I made a mistake when refering to ISO8859-1 character encoding. I meant french accented characters only.
I've fixed the AES.block_size constant issue.
I'm going to change the cipertext structure by placing the salt in front. Thanks very much.
Is Crypto.Random more appropriate than os.urandom — Emmanuel BRUNET, Dec 10 '13 at 19:58

score 1 · Accepted Answer · answered Dec 10 '13 at 13:08

1

You are doing better than I was expecting :P. Just a couple of suggestions to improve a bit your code:

Would be better if you use a nice, famous and strong key derivation function like PBKDF2 with HMAC-sha256. Your KDF looks strong, but when speaking about cryptography is better to rely on widely reviewed algorithms.
You may consider the possibility to use os.random instead of os.urandom (or at least make it simple to switch from one to another) for more entropy.
You could add some "headers" to encrypted output which would let you decrypt it without knowing before keys sizes and other variable things, that now are hard-coded.
Give the user an easier way to change those settings that now are hard-coded.

Also, for your next step, I suggest you to take a look at DH key exchange. This will give you perfect forward secrecy.

answered Dec 10 '13 at 13:08

smeso

4,165
18
27

thank you vey much for your return, I am more confident now. For HMD5 I think it's only available in python 3.3, is that correct ? – Emmanuel BRUNET Dec 10 '13 at 13:22
HMD5? Do you mean HMAC? It should be available in python 3.2.5 too. – smeso Dec 10 '13 at 13:33
Regarding the "headers" concept I'm not sure to understand but i'm going to search the web as for the DH key exchange.
For configuration purpose I use an inner class Property Property('myapp').get('AES_KeyLenght') in my own code, but I changed that in the example to improve readibility.
Best regards – Emmanuel BRUNET Dec 10 '13 at 13:34
Sorry yes it was HMAC ... ooppss. I'm going to check but it's not really a problem as I'm upgrading to Python 3.3 – Emmanuel BRUNET Dec 10 '13 at 13:35
About the headers: suppose that you and you friend don't have the same local settings about key sizes, what would happen? Nothing will work. If you implement something that will let your recipient to know what settings you have on your machine you will have no problem. – smeso Dec 10 '13 at 13:41
Ok i've got it ... Great !! – Emmanuel BRUNET Dec 10 '13 at 14:23
Using `os.random` is extremely ill advised. It may quickly deplete your entropy pool while making your code possibly *less* secure. Using a well seeded PRNG is better, using `os.urandom` is platform dependent, but OK. If you want to use DH key exchange then you also require HMAC and some kind of authentication scheme. But instead of programming a new transport protocol, I would simply deploy TLS, as generating your own transport protocol is comparable to shooting yourself in the foot. – Maarten Bodewes Dec 10 '13 at 19:32
In fact, as pointed out by @owlstead, with my suggestions and his, the requirements of your code are escalating quickly. Probably would be better to use TLS. – smeso Dec 10 '13 at 20:32
@owlstead: Why you say that os.random could make his code less secure than os.urandom? I agree with the fact that there are better source for random data than /dev/random, but I thought that /dev/urandom was worse... – smeso Dec 10 '13 at 20:37
In general, cryptographically secure PRNG's are rather secure, and if they have a big enough state, the chances of them going into a cycle are *very* slim. `/dev/urandom` moreover is periodically seeded by `/dev/random`. This seed is then mixed into the state. `/dev/random` is normally fed by OS entropy pools, e.g. small time differences in network cards or - if you are lucky - with a random generator within the CPU. Hence the number of random bytes is rather limited, and you may get into a state where more random bytes are required than available; in that case the application will *block*. – Maarten Bodewes Dec 10 '13 at 20:48
Yes, I know this. But still I don't understand why using /dev/random would be less secure than /dev/urandom. – smeso Dec 10 '13 at 20:50
It depends on the algorithm used for generating the random number. And that in turn depends on the implementation. The security part is not so much an issue though; both are reasonably secure in most implementations; the blocking part on the other hand is worse. But it is even better to use a PRNG within your application seeded from the OS using enough entropy. In that case you can generate oodles of RNG data without having to go through the kernel at all; it may of course be a good idea to reseed using the OS after generating 1MiB to 1GiB of data. – Maarten Bodewes Dec 10 '13 at 21:06
Yes, I agree that is better to use some cryptographic PRNG. – smeso Dec 10 '13 at 21:15

score 1 · Answer 2 · answered Dec 10 '13 at 19:26

Faust already made a few interesting remarks, but I have quite a few others. As Faust has already said, you seem to be heading in the right direction.

Use PBKDF2 for key derivation instead of your proprietary KDF;
Add a HMAC to the end of your ciphertext, hash your salt and any information about the algorithm as well, check the HMAC before trusting the plaintext and padding;
Please note that UTF-8 encoding is not compatible with ISO 8859-1 encoding for characters of 127 or over, it will no doubt be able to encode all the characters defined for ISO 8859-1 though;
Salt should be put in front of the ciphertext, otherwise you cannot decrypt efficiently (you will need all the ciphertext before being able to start decrypting);
You may set the IV to all zero's (and possibly not send it) if you generate a random salt each time (random IV only required if the key is re-used);
During decryption, do not rely on padding_length = padded_text[-1] blindly (see the part about HMAC);
Use constants when provided, e.g. AES.block_size instead of 16;
Use the random number generator provided by the library, you could accept anything from base class BaseRNG but use the OSRNG by default

Note that I find the RNG classes of Python crypto extremely hard to understand, keep to os.urandom if you cannot find a good way of utilizing the ones in the library.

I made a mistake when refering to ISO8859-1 character encoding. I meant french accented characters only.
I've fixed the AES.block_size constant issue.
I'm going to change the cipertext structure by placing the salt in front. Thanks very much.
Is Crypto.Random more appropriate than os.urandom — Emmanuel BRUNET, Dec 10 '13 at 20:40
Crypto.Random is more appropriate as it may be able to optimize for your application. It is also less platform dependent (e.g. /dev/urandom is not available on Windows). Choosing the right cryptographically secure RNG is less important if you don't require performance (but /dev/random may block much sooner than you may expect, so certainly don't use that). — Maarten Bodewes, Dec 11 '13 at 17:37
@EmmanuelBrunet You're welcome Emmanuel, apologies for the extended discussions about random numbers - you can easily write a book about RNG's (and people have) — Maarten Bodewes, Dec 12 '13 at 15:55

score 0 · Answer 3 · answered Dec 11 '13 at 22:03

The last source version 0.3. Hope it will help someone.

# -*- coding: utf-8 -*-
from Crypto.Cipher import AES
from Crypto.Hash import HMAC, SHA512
from pbkdf2 import PBKDF2
import os, base64, bz2, binascii

class Aes(object):
'''
Crypte / décrypte un texte donné en AES mode CBC. Accepte l'encodage base64.
Encrypts input text string & decrypts bytes encoded string with or without base64 encoding

PyCrypto and pbkdf2-1.3 packages are mandatory

Author: emmanuel.brunet@live.fr - 12/2013
''' 

SALT_LENGTH = 32 # 32 bytes = 256 bits salt
DERIVATION_ROUNDS=7000
KEY_SIZE = 32 # 256 bits key
MODE = AES.MODE_CBC

def encrypt(self, source, aes_key, outfile=None, base64_encode=False):
    '''
    Crypte l'entrée source en AES mode CBC avec sortie encodage base64 / fichier facultative

    @param str source: text to encode or text file path        
    @param bytes aes_key: password in byte
    @parm str outfile: disk file to write encoded text to. defaults to None
    @param bool base64_encode: returns base64 encoded string if True (for emails) or bytes if False

    @return bytes ciphertext: the bytes encoded string.
    '''


    '''
    ----------------------------
    Inputs management
    ----------------------------
    '''
    if os.path.exists(source):

        fp = open(source, 'rb')
        input_text = fp.read()
        fp.close()

    else:

        input_text = bytes(source, 'UTF-8')

    if input_text == b'':
        print('No data to encrypt')
        return

    '''
    # padding_len = AES.block_size - (len(input_text) % AES.block_size) 
    # padded_text = str(input_text, 'UTF-8') + chr(padding_len) * padding_len 
    '''

    '''
    -------------------
    Compress
    ------------------
    '''
    cmp_text = bz2.compress(input_text)
    b64_bin = base64.b64encode(cmp_text)
    b64_str = str(b64_bin, 'UTF-8')

    padding_len = AES.block_size - (len(b64_str) % AES.block_size) 
    padded_text = b64_str + chr(padding_len) * padding_len

    '''
    ---------------------------------------------------------
    Derived key computing PBKDF2 / specs RSA PKCS#5 V2.0 
    ---------------------------------------------------------
    '''

    salt = os.urandom(self.SALT_LENGTH)

    derived_key = PBKDF2(bytes(aes_key, 'UTF-8'), salt, iterations=self.DERIVATION_ROUNDS, digestmodule=SHA512, macmodule=HMAC).read(self.KEY_SIZE)

    '''
    ----------------
    Encrypt
    ----------------
    '''      
    # le vecteur d'initialisation doit être aléatoire
    iv = os.urandom(AES.block_size)

    Cipher = AES.new(derived_key, self.MODE, iv)
    cipher_text = Cipher.encrypt(padded_text)

    cipher_text = cipher_text + iv + salt
    # cipher_text = salt + cipher_text

    '''
    -------------------------
    Output management
    -------------------------
    '''
    if outfile is None:
        '''
        Returns cipher in base64 encoding. Useful for email management for instance
        '''
        if base64_encode:
            return(base64.b64encode(cipher_text))
        else:
            return(cipher_text)

    else:
        '''
        Writes result to disk
        '''

        fp = open(outfile, 'w')

        if base64_encode:
            fp.write(base64.b64encode(cipher_text))
        else:
            fp.write(cipher_text)

        fp.close()

        print('Cipher text saved in', outfile)


def decrypt(self, source, aes_key, outfile=None, base64_encode=False):
    '''
    @param bytes or str source: encrypted bytes string to decode or file path        
    @param bytes aes_key: password
    @parm str outfile: disk file to write encoded text to. defaults to None        
    @param bool base64_encode: cipher text is given base64 encoded (for mails content for examples)

    @returns str secret_text: the decoding text string or None if invalid key given
    '''

    '''
    ---------------------------
    Input management
    ---------------------------
    '''

    if type(source) == str and os.path.exists(source):

        fp = open(source, 'rb')
        ciphertext = fp.read()
        fp.close()

    elif type(source) == bytes:

        ciphertext = source

    else:
        print('Invalid data source')
        return

    if base64_encode:

        encoded_text = base64.b64decode(ciphertext)

    else:

        encoded_text = ciphertext

    salt_start = len(encoded_text) - self.SALT_LENGTH
    iv_start = len(encoded_text) - AES.block_size - self.SALT_LENGTH        
    data, iv, salt = encoded_text[:iv_start], encoded_text[iv_start:salt_start], encoded_text[salt_start:]

    '''
    -------------------------
    Derived key computing
    -------------------------
    '''

    # derived_key = PBKDF2(bytes(aes_key, 'UTF-8'), salt).read(self.KEY_SIZE)
    derived_key = PBKDF2(bytes(aes_key, 'UTF-8'), salt, iterations=self.DERIVATION_ROUNDS, digestmodule=SHA512, macmodule=HMAC).read(self.KEY_SIZE)   

    '''
    -------------------------
    Decrypt
    -------------------------
    '''
    Cipher = AES.new(derived_key, self.MODE, iv)
    padded_text = Cipher.decrypt(data)

    padding_length = padded_text[-1]
    secret_text = padded_text[:-padding_length]

    '''
    --------------------------
    Decompress
    --------------------------
    '''

    cmp_text = base64.b64decode(secret_text)
    secret_text = bz2.decompress(cmp_text)

    '''
    Si le flux n'est pas décodé (mot de passe invalide),  la conversion UTF-8 plante ou au mieux on obtient un texte illisible
    '''
    try:
        secret_text = str(secret_text, 'utf-8')
    except:
        return

    if outfile is None:

        return(secret_text)

    else:
        '''
        Writes result to disk
        '''
        fp = open(outfile, 'w')
        fp.write(secret_text)
        fp.close()

Python 3.2 - Symétric AES Crypto in CBC mode - advices requested

3 Answers3

Linked