What's the fastest way to compare bits of a hash in Python3?

Question

I'm looking to compare bits of a hash in Python3, as part of a Hashcash system. So for instance, I want to know if the first N bits of a SHA256 hash are 0.

Right now, I'm doing this based on the hex version

  if newhash.hexdigest()[0:4] == '0000'

But this doesn't let me be as granular as I want - I'd prefer to compare the raw bits, which lets me vary the number of matching 0s far more closely.

I get get the bit values to compare through a convoluted hop

  bin(int(h.hexdigest(), 16))[2:]

but this seems like it can't possibly be the fastest/right way to do it.

I'd appreciate any advise on the right/correct way to do it ;)

Thanks,

-CPD

Counting leading zeros ([finding most significant bit set](http://graphics.stanford.edu/~seander/bithacks.html#IntegerLogObvious)) can be optimized relative to a bit comparison in general. — jfs, Mar 26 '13 at 18:01

user4815162342 · Answer 1 · 2013-03-30T09:16:10.873

To check whether selected bits of a number are zero, you need to and the number with a precomputed mask that has all those bits set, and compare the result to zero. The mask that checks for the first n bits of an m-bit number is the number that consists of n 1s followed by m - n 0s in binary.

def mask(n, m):
    return ((1 << n) - 1) << (m - n)

def test_0bits(digest_bytes, n_bits):
    m = 8 * len(digest_bytes)
    digest_num = int.from_bytes(digest_bytes, 'big')
    return digest_num & mask(n_bits, m) == 0

>>> test_0bits(b'\123\456', 3)  # 001 010 011 100 101 110
False
>>> test_0bits(b'\023\456', 3)  # 000 010 011 100 101 110
True

If you keep calling test_bits with the same number of bits, you can precompute the mask and store it as a module-level "constant".

score 0 · Answer 2 · answered Mar 26 '13 at 16:07

0

You can get the first 8 bytes of the digest unpacked like this:

bin(struct.unpack('>Q', h.digest()[:8])[0])

but I'm not sure if it's faster, and it won't be convenient for the rest of the bits. Bit-twiddling is not easy in Python.

answered Mar 26 '13 at 16:07

Ned Batchelder

364,293
75
561
662

score 0 · Answer 3 · answered Mar 26 '13 at 16:11

If you can deal with indexing bits from the right, the integer type in gmpy2 supports slicing to access individual bits:

>>> x=gmpy2.mpz(12345)
>>> x.digits(2)
'11000000111001'
>>> x[2:5].digits(2)
'110'

If you need to modify individual bits, gmpy2 includes a mutable integer type that allows you to modify the bits in place.

Disclaimer: I maintain gmpy2.

What's the fastest way to compare bits of a hash in Python3?

3 Answers3