Why does checking if a HashMap has a certain value take very long to execute within a for loop?

Question

I am working with meet-in-the-middle attack on Double-DES. I have implemented the DES encryption/decryption and done the encryptions and now I would like to perform a MITM attack on the Double-DES in order to find the keys. The way I am trying to achieve this is by storing, within a for loop, the intermediate ciphers as the key of a HashMap and the possible keys as the values of the HashMap. However, within this for loop I also want to make sure that the possible keys are unique i.e. I have an if statement, which checks whether the possible key already exists in the HashMap. If not, then it uses it to encrypt the plain text and store the cipher text and the possible key in the HashMap. Aftwards, I try to find the keys, which have matching intermediate cipher text by iterating through the HashMap with a foreach and compare each intermediate cipher text from the encryptions with the intermediate cipher text from the decryptions.

However, I am unable to find the match as it takes too long to finish. I have been waiting for like 2 hours now without any result. If I remove the if statement, which checks if the possible key is already in the HashMap it finishes in like 10 seconds.

for (int size = intermediateCipher.size(); size < Math.pow(2, 20); size++) { // intermediateCipher is my HashMap consisting of <String, byte[]>
    byte[] possibleKey = generateDesKey(); // generateDesKey generates a key of 64 bits
    if (!intermediateCipher.containsValue(possibleKey)) {
        intermediateCipher.put((encrypt(possibleKey, plainText)).toString(), possibleKey);
    }
}

int count = 0;
for (Entry<String, byte[]> arr : intermediateCipher.entrySet()) {

    String temp = (decrypt(arr.getValue(), cipherText)).toString();

    if (intermediateCipher.containsKey(temp)) {
        count++;
    }
}

I should mention that only 20 bits of the DES key is effective. That is why there are 2^20 possible keys. Furthermore, if I don't have the if statement, which checks if the possible key is already in the HashMap, I get 510 matches, which is too much.

UPDATE:

I have tried using a Set in order to store the keys first and then used the keys from the Set to encrypt etc. However, instead of having a for, which iterates from 0 to 2^20, I have tried with a while loop, which checks iterates as long as the Set has elements. However, I have tried running this approach for over 10 mins without any result. It never gets out of the loop.

for (int i = 0; i < Math.pow(2, 20); i++) {
            possibleKeySet.add(generateDesKey());
        }
        System.out.println(possibleKeySet.size());

        for (int i = 0; i < possibleKeySet.size(); i++) {

            intermediateCipher.put((encrypt(possibleKeySet.iterator().next(), plainText)).toString(),
                    possibleKeySet.iterator().next());
        }

        System.out.println("ss " + intermediateCipher.size());

        int count = 0;
        for (Entry<String, byte[]> arr : intermediateCipher.entrySet()) {

            String temp = ((decrypt(arr.getValue(), cipherText)).toString());
            if (intermediateCipher.containsKey(temp)) {
                count++;
            }
        }

I have read that for a set hasNext() always returns true for a non-empty collection. So, I have tried with a for each, but the size of the hashMap is never the same as the size of the set of keys, which doesn't make sense to me as I use every key in the set:

for (int i = 0; i < Math.pow(2, 20); i++) {
            possibleKeySet.add(generateDesKey());
        }

        System.out.println(possibleKeySet.size());

        for (byte[] key : possibleKeySet) {
            intermediateCipher.put((encrypt(key, plainText)).toString(),
                    key);
        }

if `temp` isn't used, you don't need to `toString` which means you don't need to decrypt, which means you don't need to iterate the Map. In short, the JVM is pretty good at eliminating code which doesn't do anything. — Peter Lawrey, Nov 22 '18 at 17:55
I think this is one of those naive attacks that will take approximately seven times as long as the heat death of the universe. You need something like [Deep Crack](https://en.wikipedia.org/wiki/EFF_DES_cracker) for DES. — Elliott Frisch, Nov 22 '18 at 17:55
@PeterLawrey: What do you mean? I am using it in the next if statement or do you mean something else? — Bab, Nov 22 '18 at 17:57
@Bab I assumed if you removed the `if` you removed the `intermediateCipher.containsKey(temp)` but again, without the `if` this method call doesn't do anything either. — Peter Lawrey, Nov 22 '18 at 17:58
The `if` statement he's talking about appears to be `if (!intermediateCipher.containsValue(possibleKey))`, and the key is not a *map* key, but rather a cryptographic key stored as a map *value*. — John Bollinger, Nov 22 '18 at 18:02

score 1 · Answer 1 · answered Nov 22 '18 at 18:08

1

Map.containsValue() will have a linear time as you are iterating blindly over all the map values. As per the method javadoc:

Returns true if this map maps one or more keys to the specified value. More formally, returns true if and only if this map contains at least one mapping to a value v such that Objects.equals(value, v). This operation will probably require time linear in the map size for most implementations of the Map interface.

To take advantage of a hash lookup you should check the keys with Map.containsKey():

if (!intermediateCipher.containsKey(possibleKey)) {
  ...
}

answered Nov 22 '18 at 18:08

Karol Dowbecki

43,645
9
78
111

So, I should change my HashMap to as the possible key is stored as the value now? – Bab Nov 22 '18 at 18:12
@Bab If your lookup is the primary cost and you have enough memory to maintain the Map, yes. Or use a separate `Set` for just checking presence of `possibleKey`. – Karol Dowbecki Nov 22 '18 at 18:13
Have tried changing the HashMap, but it seems that it doesn't help. I am going for the Set as it doesn't allow duplicates. However, I would like to ask, if I choose this approach, I need to have a for loop, which generates, in my case, 2^20 keys, right? I can afterwards use these keys as an element (key/value?) in the HashMap? – Bab Nov 22 '18 at 18:31
Have tried with `while (possibleKeySet.iterator().hasNext()) { intermediateCipher.put((encrypt(possibleKeySet.iterator().next(), plainText)).toString(), possibleKeySet.iterator().next()); }` – Bab Nov 22 '18 at 19:29
Have been waiting for over 10 mins. – Bab Nov 22 '18 at 19:30
@Bab what is `intermediateCipher.size()` value? – Karol Dowbecki Nov 22 '18 at 19:38
As it doesn't finish I am unable to print it out :/ I tried to print the size at the end i.e. after my for loop with the decryptions. Have also tried after the while loop, but it didn't give finish, so I didn't get a result. Furthermore, I have also tried with 2^2 instead of 2^20. – Bab Nov 22 '18 at 19:40
@Bab Please provide a [minimum verifiable example](https://stackoverflow.com/help/mcve). – Karol Dowbecki Nov 22 '18 at 20:13
@Bab Your code doesn't even compile mate. I can guess some of the missing pieces but there is no way to tell what does `encrypt()` and `decrypt()` does. Also `possibleKeySet.iterator().next()` makes little sense as it always returns the same key. Why are you always using the same key if you generated 2^20 of them? – Karol Dowbecki Nov 22 '18 at 20:58
As I am not supposed to post all my code, I did describe what encrypt/decrypt is in the OP :) Anyway, as I figured out that `possibleKeySet.iterator().next()` returns the same key (I actually tried to print it), I changed over to a for each as my while loop never ended, I have read that for a set `hasNext()` always will return true for a non-empty collection. With that being said, I did try with a for each, but the size of the hashMap is never the same as the size of the set of keys, which doesn't make sense to me as I use every key in the set – Bab Nov 22 '18 at 21:07
@Bab you don't have to give away your code, just provide an example that compiles, runs and reproduces the problem. Remove the pieces that are not important but make sure it has a working `main()`. – Karol Dowbecki Nov 22 '18 at 21:08

score 1 · Answer 2 · answered Nov 22 '18 at 18:10

1

Map.containsValue() must check every map entry. That costs O(n) in the size of the map. Supposing that the number of duplicates does not exceed a fixed fraction of all keys generated, checking containsValue() on each iteration of your loop then costs an aggregate O(n²) in the number of keys generated. For a million keys that's awfully expensive.

Consider instead keeping an auxiliary Set of the keys stored so far, and testing membership in that set instead of directly testing the values in the Map.

answered Nov 22 '18 at 18:10

John Bollinger

160,171
8
81
157

By using a Set, I need to have a for loop, which generates, in my case, 2^20 keys, right? I can afterwards use these keys as an element (key/value?) in the HashMap? – Bab Nov 22 '18 at 18:32
Have tried with a set, as you suggested, where I store all the possible keys. Afterwards, I run `while (possibleKeySet.iterator().hasNext()) { intermediateCipher.put((encrypt(possibleKeySet.iterator().next(), plainText)).toString(), possibleKeySet.iterator().next()); }` but that didn't help either i.e. have been waiting for over 10 mins. Is that normal? – Bab Nov 22 '18 at 19:33

Why does checking if a HashMap has a certain value take very long to execute within a for loop?

2 Answers2

Linked