Questions tagged [crc32]

A cyclic redundancy check (CRC) is an error-detecting code designed to detect accidental changes to raw computer data, and is commonly used in digital networks. (wiki) A CRC32 algorithm typically takes in a file stream or character array and calculates an unsigned long codeword from the input. One can transmit this codeword and re-calculate it on the receiver end, then compare it to the transmitted one to detect an error.

A cyclic redundancy check (CRC) is an error-detecting code commonly used in digital networks and storage devices to detect accidental changes to raw data. Blocks of data entering these systems get a short check value attached, based on the remainder of a polynomial division of their contents; on retrieval the calculation is repeated, and corrective action can be taken against presumed data corruption if the check values do not match. wikipedia

The most commonly used polynomial lengths are:

  • CRC-8: 9 bits
  • CRC-16: 17 bits
  • CRC-32: 33 bits
  • CRC-64: 65 bits

A truly excellent tutorial on CRC's is Ross Williams' "Painless Guide to CRC Detection Algorithms", which can also be found here, here, here, here, and here.

570 questions
7
votes
6 answers

the fastest way to create checksum for large files in python

i need to transfer large files across network and need to create checksum for them on hourly basis. so the speed for generating checksum is critical for me. somehow i can't make zlib.crc32 and zlib.adler32 working with files larger than 4GB on…
pixelblender
  • 181
  • 3
  • 4
  • 8
7
votes
1 answer

Speed up my indexes in MySQL - CRC or MD5?

I've got a huge table with something like 8 300 000 rows (won't be edit nor delete ever). My first column look something similar P300-4312B_X16_S and the entry isn't unique so I use a regular INDEX on this field. However, MySQL is WAY faster using a…
David Bélanger
  • 7,400
  • 4
  • 37
  • 55
7
votes
1 answer

XAPK File validation shows in correct data Information

While running APK expansion file sample I just skiped checking CRC32 to avoide crc bug algorithm and its working very fine!! But while XAPK File validation shows in correct downloaded data information 99% !!, how to avoid that and view full 100% by…
LOG_TAG
  • 19,894
  • 12
  • 72
  • 105
6
votes
1 answer

What is a running CRC?

I have searched and am not able to find information on what it is and how it is computed. I have no idea why the question has been negative voted. Is it not clear and programming related? Or should I have asked: # Or you can compute the running…
Alan Haggai Alavi
  • 72,802
  • 19
  • 102
  • 127
6
votes
4 answers

CRC-32 implementation in java.util.zip.CRC32

Which CRC-32 algorithm is used in the Java CRC-32 class ? The java doc does not give any details. What is the polynomail used and the initial value for calculaton ?
Dunxton
  • 408
  • 1
  • 8
  • 21
6
votes
0 answers

Java: update CRC32 from a stream

Am I misunderstanding the use of either of the CRC32 or CheckedInputStream classes to calculate a checksum by continuously updating with the latest input? When the input is <= 128KiB a valid CRC32 is generated. Anything larger than 128KiB and the…
kaveman
  • 4,339
  • 25
  • 44
6
votes
1 answer

Fast numeric hash function for Spark (PySpark)

I am trying to apply a hash function to short strings in a column of a PySpark DataFrame (running on an EMR cluster) and get a numeric value as a new column. CRC3 would do the job for example. I am aware of this question, but it's in Scala, I need a…
Alt
  • 2,597
  • 5
  • 26
  • 36
6
votes
1 answer

What would a big-endian compatible version of this CRC32 method look like?

I'm working on a project that requires a CRC32 check to be done on data that is being transmitted. I would like to make my code compatible for not only Intel architecture ("Little Endian"), but for Solaris architecture ("Big Endian") as well. I've…
jiman
  • 270
  • 1
  • 3
  • 8
6
votes
1 answer

How to check CRC32 of NSData?

Possible Duplicate: Get CRC checksum of an NSData in Objective-C I can't find any implementation of CRC32 algoryghm in xcode. Can anybody help me calculate it?
Funky Kat
  • 61
  • 1
  • 4
6
votes
2 answers

Difference between CRC and hash method (MD5, SHA1)

Both CRC and hash methods can be used to verify the integrity of the original data. Why do most systems uses hash method nowadays?
user496949
  • 83,087
  • 147
  • 309
  • 426
6
votes
2 answers

Python CRC-32 woes

I'm writing a Python program to extract data from the middle of a 6 GB bz2 file. A bzip2 file is made up of independently decryptable blocks of data, so I only need to find a block (they are delimited by magic bits), then create a temporary…
Lauritz V. Thaulow
  • 49,139
  • 12
  • 73
  • 92
6
votes
1 answer

Fixing "crc32 tool not found" on Ubuntu

I'm trying to build spark/firmware on Ubuntu, but when running make I only get the following error message ../build/common-tools.mk:26: *** "crc32 tool is not found". Stop. I tried using apt-get install crc32 but there is no such package. How can…
Uli Köhler
  • 13,012
  • 16
  • 70
  • 120
6
votes
2 answers

Calculate a CRC / CRC32 hash / checksum on a binary file in Python using a buffer

I've been trying to teach myself Python so I don't fully understand what I'm doing. I'm embarrassed to say this but my question should be really easy to answer. I want to be able to do a CRC checksums on binary files with code similar to this: #…
Dave Brunker
  • 1,559
  • 5
  • 15
  • 23
6
votes
3 answers

Test vectors for CRC32C

I'm writing test harness for an CRC calculation library and I'm looking for reference test vectors for CRC-32C. I found plenty for CRC-32 but nothing for CRC-32C specifically. Could somebody point me to a reference? I managed to calculate these…
dtoux
  • 1,754
  • 3
  • 21
  • 38
5
votes
4 answers

Why is this CRC32 implementation in C# so slow?

I'm using the following function to compute the CRC32 of a file in a VS2008, .NET 3.5 project: public UInt32 ComputeHash(System.IO.Stream stream) { unchecked { const int BUFFER_SIZE = 1024; UInt32 crc32Result = 0xFFFFFFFF; …
raven
  • 18,004
  • 16
  • 81
  • 112