How to find crc32 of big files?

Question

The PHP's crc32 support string as input.And For a file , below code will work OFC.

crc32(file_get_contents("myfile.CSV"));

But if file goes huge (2 GB) it might raise out of memory Fatal error.

So any way around to find checksum of huge files ?

Answer 1

PHP doesn't support files larger than 2GB (32bit limitation)

And more efficient way to calculate crc32 from files:

$hash = hash_file('crc32b',"myfile.CSV" );

Answer 2

This function in the User Contributed Notes to crc32() claims to calculate the value without loading the file in full. If it works correctly, it should eliminate any memory problems.

For a file larger than 2 GB, it is however likely to stop at the same 32-bit limitation you are encountering right now.

If possible, I would invoke an external tool that can calculate the checksum for files as large as the one at hand.

Answer 3

dev-null-dweller's answer is IMO the way to go.

However, for those who are looking for a memory-efficient PHP4 backport of hash_file('crc32b', $filename);, here is a solution based on this PHP manual comment, with some improvements:

• It now gives exactly the same results than hash_file()
• It supports 32 bit & 64 bit architectures.

Warning: perfs are ugly. Trying to improve.

Note: I've tried a solution based on the C source code from zaf's comment, but I could not quickly enough succeed to port it to PHP.

if (!function_exists('hash_file'))
{
    define('CRC_BUFFER_SIZE', 8192);

    function hash_file($algo, $filename, $rawOutput = false)
    {
        $mask32bit = 0xffffffff;

        if ($algo !== 'crc32b')
        {
            trigger_error("Unsupported hashing algorightm '".$algo."'", E_USER_ERROR);
            exit;
        }

        $fp = fopen($filename, 'rb');

        if ($fp === false)
        {
            trigger_error("Could not open file '".$filename."' for reading.", E_USER_ERROR);
            exit;
        }

        static $CRC32Table, $Reflect8Table;
        if (!isset($CRC32Table))
        {
            $Polynomial = 0x04c11db7;
            $topBit = 1 << 31;

            for($i = 0; $i < 256; $i++)
            {
                $remainder = $i << 24;
                for ($j = 0; $j < 8; $j++)
                {
                    if ($remainder & $topBit)
                        $remainder = ($remainder << 1) ^ $Polynomial;
                    else
                        $remainder = $remainder << 1;

                    $remainder &= $mask32bit;
                }

                $CRC32Table[$i] = $remainder;

                if (isset($Reflect8Table[$i]))
                    continue;
                $str = str_pad(decbin($i), 8, '0', STR_PAD_LEFT);
                $num = bindec(strrev($str));
                $Reflect8Table[$i] = $num;
                $Reflect8Table[$num] = $i;
            }
        }

        $remainder = 0xffffffff;
        while (!feof($fp))
        {
            $data = fread($fp, CRC_BUFFER_SIZE);
            $len = strlen($data);
            for ($i = 0; $i < $len; $i++)
            {
                $byte = $Reflect8Table[ord($data[$i])];
                $index = (($remainder >> 24) & 0xff) ^ $byte;
                $crc = $CRC32Table[$index];
                $remainder = (($remainder << 8) ^ $crc) & $mask32bit;
            }
        }

        $str = decbin($remainder);
        $str = str_pad($str, 32, '0', STR_PAD_LEFT);
        $remainder = bindec(strrev($str));
        $result = $remainder ^ 0xffffffff;
        return $rawOutput ? strrev(pack('V', $result)) : dechex($result);
    }
}