What is wrong in my RC6 implmentation?

Question

can anyone see where i made a mistake here? I know that the algorithm will properly decrypt the encrypted data. however, most of the encrypted data is not the correct output, according to the RC6 paper.

// hexlify(string) turns a string into its hex representation: hexlify("AB") -> "4142"
// unhexlify(string) turns a string into its ASCII representation: unhexlify("4142") -> "AB"
// uint128_t is my own version of uint128, and Im pretty sure that the math is correct
// little_end(string, base) flips a string by bytes to get the little endian version of the string 
// ROL/ROR(int, rotate x bits, bitsize of input int) does bitwise rotation


class RC6{
    private:
        unsigned int w, r, b, lgw;
        std::vector <uint32_t> S;
        uint128_t mod;
        std::string mode;

        void keygen(std::string KEY){
            uint64_t p, q;
            rc_pq(w, p, q);
            KEY = hexlify(KEY);
            unsigned int u = (unsigned int) ceil(w / 8.);
            unsigned int c = (unsigned int) ceil(float(b) / u);
            while ((KEY.size() >> 1) % u != 0)
                KEY += zero;
            std::vector <uint32_t> L;
            for(unsigned int x = 0; x < c; x++)
                L.push_back(toint(little_end(KEY.substr(2 * u * x, 2 * u), 16), 16));
            S.push_back(p);
            for(unsigned int i = 0; i < 2 * r + 3; i++)
                S.push_back((S[i] + q) % mod);
            uint32_t A = 0, B = 0, i = 0, j = 0;
            uint32_t v = 3 * std::max(c, 2 * r + 4);
            for(unsigned int s = 1; s < v + 1; s++){
                A = S[i] = ROL((S[i] + A + B) % mod, 3, w);
                B = L[j] = ROL((L[j] + A + B) % mod, (A + B) % w, w);
                i = (i + 1) % (2 * r + 4);
                j = (j + 1) % c;
            }
        }

    public:
        RC6(std::string KEY, std::string MODE, unsigned int W = 32, unsigned int R = 20, unsigned int B = 16){
            w = W;
            r = R;
            b = B;
            mod = uint128_t(1) << w;
            lgw = (unsigned int) log2(w);
            mode = MODE;
            keygen(KEY);
        }

        std::string run(std::string DATA){
            DATA = hexlify(DATA);
            uint32_t A = toint(little_end(DATA.substr(0, 8), 16), 16), B = toint(little_end(DATA.substr(8, 8), 16), 16), C = toint(little_end(DATA.substr(16, 8), 16), 16), D = toint(little_end(DATA.substr(24, 8), 16), 16);
            if (mode == "e"){
                B += S[0];
                D += S[1];
                for(unsigned int i = 1; i < r + 1; i++){
                    uint64_t t = ROL((uint64_t) ((B * (2 * B + 1)) % mod), lgw, w);
                    uint64_t u = ROL((uint64_t) ((D * (2 * D + 1)) % mod), lgw, w);
                    A = ROL(A ^ t, u % w, w) + S[2 * i];
                    C = ROL(C ^ u, t % w, w) + S[2 * i + 1];
                    uint64_t temp = A; A = B % mod; B = C % mod; C = D % mod; D = temp % mod;
                }
                A += S[2 * r + 2];
                C += S[2 * r + 3];
            }
            else{
                C -= S[2 * r + 3];
                A -= S[2 * r + 2];
                for(int i = r; i > 0; i--){
                    uint64_t temp = D; D = C % mod; C = B % mod; B = A % mod; A = temp % mod;
                    uint64_t u = ROL((uint64_t) ((D * (2 * D + 1)) % mod), lgw, w);
                    uint64_t t = ROL((uint64_t) ((B * (2 * B + 1)) % mod), lgw, w);
                    C = ROR((C - S[2 * i + 1]) % mod, t % w, w) ^ u;
                    A = ROR((A - S[2 * i]) % mod, u % w, w) ^ t;
                }
                D -= S[1];
                B -= S[0];
            }
            w >>= 2;
            return unhexlify(little_end(makehex(A % mod, w)) + little_end(makehex(B % mod, w)) + little_end(makehex(C % mod, w)) + little_end(makehex(D % mod, w)));
        }
};

of these tests vectors, only the first two are correct. the rest are not

data = "00000000000000000000000000000000";
key = "00000000000000000000000000000000";
ciphertext = "8fc3a53656b1f778c129df4e9848a41e";

data = "02132435465768798a9bacbdcedfe0f1";
key = "0123456789abcdef0112233445566778";
ciphertext = "524e192f4715c6231f51f6367ea43f18";


data = "00000000000000000000000000000000";
key = "000000000000000000000000000000000000000000000000";
ciphertext = "6cd61bcb190b30384e8a3f168690ae82";


data = "02132435465768798a9bacbdcedfe0f1";
key = "0123456789abcdef0112233445566778899aabbccddeeff0";
ciphertext = "688329d019e505041e52e92af95291d4";


data = "00000000000000000000000000000000";
key = "0000000000000000000000000000000000000000000000000000000000000000";
ciphertext = "8f5fbd0510d15fa893fa3fda6e857ec2";


data = "02132435465768798a9bacbdcedfe0f1";
key = "0123456789abcdef0112233445566778899aabbccddeeff01032547698badcfe";
ciphertext = "c8241816f0d7e48920ad16a1674e5d48";

did i mess up a uint somewhere? wrong little endian change?

I like it that the code is close to the paper's pseudcode. To proceed I suggest you write unit tests for uint128_t and the other primitive types/functions you wrote to make sure they're working as advertized. — , Jun 05 '11 at 21:54
i have tested uint128_t quite a bit. there may be stuff to fix, but i dont believe that it will be relevant to this code. it can be changed to uint64_t if you wish. the others have also been extensively tested and found to work properly — calccrypto, Jun 05 '11 at 22:07
If these are the ciphertexts you're getting they seem identical to the paper's ciphertexts to me - I take it your code isn't giving you these values? — , Jun 05 '11 at 23:42
correct. it doesnt make sense. im getting the same values in my python version of this too — calccrypto, Jun 06 '11 at 05:36
I'd check your code agianst some other open source RC6 implementation. If you debug them side by side, you may see the source of the problem. I found one here: http://www.codeproject.com/KB/security/hexenc.aspx — , Jun 06 '11 at 09:16
their code is such a mess. most of it involves dealing with windows. I couldnt find the part that actually did the encryption. also, why in the world is their password entry method `type a number between 1 and 1000 000 000`????? that makes no sense whatsoever — calccrypto, Jun 06 '11 at 18:18
For the test vectors you don't make clear what the expected and what the actual values produced by your code are. — starblue, Jun 07 '11 at 07:14
Don't use `unsigned int`, always use types of specific size. — starblue, Jun 07 '11 at 07:15
What do you mean not clear??? The test vectors are what I expect as outputs. I am not getting them. When I encrypt example 3 data with the key, I don't get the ciphertext. I have no intermediate values to compare them to, so I don't know what is wrong. Thus it doesn't matter what I'm getting, just not the right values — calccrypto, Jun 07 '11 at 08:36

score 0 · Accepted Answer · answered Jun 07 '11 at 08:59

0

I think I figured it out. Can anyone corroborate? I think that because I set b = 16 by default, I'm causing the errors. My harddrive is dead or I would have tested this already

answered Jun 07 '11 at 08:59

calccrypto

8,583
21
68
99

1

_b_ is supposed to be the key length, in bytes. It should be 16 when you use a 16-byte key (as in your two first test vectors), 24 then the key consists in 24 bytes (your third and fourth vectors), and so on. – Thomas Pornin Jun 07 '11 at 13:30
Yep. Just tested it. It works now! Finally! Why couldnt someone just tell me?? – calccrypto Jun 07 '11 at 20:25

What is wrong in my RC6 implmentation?

1 Answers1