-1

I want to compare a fix-size buffer of 128MB with checksum with C language. I used sha1 to get a 20 bytes digest which cost 1700ms in my centos box. It there any algorithm would that satisfy my requirement and cost less then 50ms?

George_BJ
  • 596
  • 2
  • 9
  • 21
  • MD5 might be faster, but it is as insecure as SHA1. Only use MD5 or SHA1 if you really know what you are doing! – idmean Jul 31 '15 at 09:18
  • Do you need cryptographic properties for your checksum or is it just for consistency/error-checking purposes? – rohit89 Jul 31 '15 at 09:21
  • @rohit89 just for consistency, no need for crypt – George_BJ Jul 31 '15 at 09:36
  • 1
    Maybe you can find CRC implementation that'll suit your needs. https://en.wikipedia.org/wiki/Cyclic_redundancy_check – Agent_L Jul 31 '15 at 09:44
  • 100x times faster than SHA1 might well exceed sequential memory speed. Here that would be about 6500MB/sec hash throughput. – usr Jul 31 '15 at 10:15
  • I tried https://github.com/mattsta/crcspeed which took about 330ms, I also tried adler32 which only took about 170ms. But I am not sure if the collision rate of adler32 is capable for 128MB data. – George_BJ Aug 03 '15 at 07:33

2 Answers2

2

Simply do a one integer XOR sum:

int xorsum = 0;
for(int i=0; i< length; i++) xorsum ^= mem[i];

Here your memory is considered an int array.

For higher "precision" you could do the same with a xor-sum of length 2,3 or more integers.

DrKoch
  • 9,556
  • 2
  • 34
  • 43
  • I takes about 200ms using your suggested xor sum. – George_BJ Aug 04 '15 at 02:56
  • If you need it faster you could sacrifice some _precision_ and take every 2nd or 4th memory position only. – DrKoch Aug 04 '15 at 04:57
  • Did you try processing 4 or 8 bytes at a time, instead of one byte at a time? That is, not skip bytes, but cast to long pointer, or long long pointer. If you do that, I think even Fletcher32 should be fast enough, according to my experiment. – Thomas Mueller Aug 07 '15 at 09:00
2

Say you are asking for algorythm and not library, because the latter would be off topic ...

You could try to use Cyclic Redundancy Check. There do not address cryptography so they are easier to compute.

You will find references on Wikipedia and nice C implementations at http://www.lammertbies.nl with an online CRC calculator to control your implementation.

Serge Ballesta
  • 143,923
  • 11
  • 122
  • 252