Java, make murmur2 hash work on portion of byte array

Question

I'm using murmur2 hash on a byte array but I only want to hash a subset of the bytes, murmur2 only lets me hash the array starting from 0 but I want to specify a non 0 start offset as well as an end offset in the array.

     * 
 * @param data byte array to hash
 * @param length length of the array to hash
 * @param seed initial seed value
 * @return 32 bit hash of the given array
 */
public static int hash32(final byte[] data, int length, int seed) {
    // 'm' and 'r' are mixing constants generated offline.
    // They're not really 'magic', they just happen to work well.
    final int m = 0x5bd1e995;
    final int r = 24;

    // Initialize the hash to a random value
    int h = seed^length;
    int length4 = length/4;

    for (int i=0; i<length4; i++) {
        final int i4 = i*4;
        int k = (data[i4+0]&0xff) +((data[i4+1]&0xff)<<8)
                +((data[i4+2]&0xff)<<16) +((data[i4+3]&0xff)<<24);
        k *= m;
        k ^= k >>> r;
                k *= m;
                h *= m;
                h ^= k;
    }

    // Handle the last few bytes of the input array
    switch (length%4) {
    case 3: h ^= (data[(length&~3) +2]&0xff) << 16;
    case 2: h ^= (data[(length&~3) +1]&0xff) << 8;
    case 1: h ^= (data[length&~3]&0xff);
    h *= m;
    }

    h ^= h >>> 13;
    h *= m;
    h ^= h >>> 15;

                return h;
}

I have tried various changes but it always causes my hash collisions test to go from 0 to a very high number. I don't want to use murmur3 because it does not fit in a single small method like murmur2, murmur2 is also a little faster in my tests.

Here's my collisions tester for anyone who want to have a crack at it

            HashSet<Integer> hs = new HashSet<>(100000000,(float) 1.0);
        long collide = 0;
        long totalLoops = 0;
        byte[] ba = new byte[4];
        long sTime = System.currentTimeMillis();
        int hash;
        for(byte d=0; d<5; d++) {
            ba[0] = d;
        for(byte i=-128; i<127; i++) {
            ba[1] = i;
            for(byte k=-128; k<127; k++) {
                ba[2] = k;
            for(byte j=-128; j<127; j++) {
                ba[3] = j;
                hash = hash32(ba,ba.length,0x9747b28c);
                if(hs.contains(hash)) {
                    collide++;
                } else {
                    hs.add(hash);
                }
                totalLoops++;
            }
            }
        }
        }

Note: the collision test above needs a pc with 8GB of RAM.

score 0 · Accepted Answer · answered Aug 10 '15 at 23:44

I found a murmurhash3 implementation that uses an offset for those interested, problem solved.

public static int murmurhash3_x86_32(byte[] data, int offset, int len, int seed) {

final int c1 = 0xcc9e2d51;
final int c2 = 0x1b873593;

int h1 = seed;
int roundedEnd = offset + (len & 0xfffffffc);  // round down to 4 byte block

for (int i=offset; i<roundedEnd; i+=4) {
  // little endian load order
  int k1 = (data[i] & 0xff) | ((data[i+1] & 0xff) << 8) | ((data[i+2] & 0xff) << 16) | (data[i+3] << 24);
  k1 *= c1;
  k1 = (k1 << 15) | (k1 >>> 17);  // ROTL32(k1,15);
  k1 *= c2;

  h1 ^= k1;
  h1 = (h1 << 13) | (h1 >>> 19);  // ROTL32(h1,13);
  h1 = h1*5+0xe6546b64;
}

// tail
int k1 = 0;

switch(len & 0x03) {
  case 3:
    k1 = (data[roundedEnd + 2] & 0xff) << 16;
    // fallthrough
  case 2:
    k1 |= (data[roundedEnd + 1] & 0xff) << 8;
    // fallthrough
  case 1:
    k1 |= (data[roundedEnd] & 0xff);
    k1 *= c1;
    k1 = (k1 << 15) | (k1 >>> 17);  // ROTL32(k1,15);
    k1 *= c2;
    h1 ^= k1;
}

// finalization
h1 ^= len;

// fmix(h1);
h1 ^= h1 >>> 16;
h1 *= 0x85ebca6b;
h1 ^= h1 >>> 13;
h1 *= 0xc2b2ae35;
h1 ^= h1 >>> 16;

return h1;

}

Java, make murmur2 hash work on portion of byte array

1 Answers1