How can I sum large hexadecimal values in Perl?

Question

I'm iterating over a set of 32-bit hexadecimal strings ("DEADBEEF", "12345678", etc..) and I'm trying to sum them together to form a 32-bit checksum. Assume that the variable $temp is loaded with some hexadecimal string in the example below.

my $temp;
my $checksum;

for (...)
{
    #assume $temp is loaded with a new hex string here
    my $tempNum = hex ($temp);
    $checksum += $tempNum;
    $checksum &= 0xFFFFFFFF;
    print printf("checksum: %08X",$checksum);
}

The first few values are "7800798C", "44444444", and "44444444". The output is:

checksum: 7800798C
checksum: BC44BDD0
checksum: FFFFFFFF
checksum: FFFFFFFF

etc..

as you can see the first two summations are correct and then it seems to saturate. Am I missing something regarding the size limit of Perl variables?

EDIT: This is the actual output from the script (string is the hex string, value is the decimal conversion of that string, and checksum is the resulting output):

string: 7800798C, value: 2013297036, checksum 7800798C
string: 44444444, value: 1145324612, checksum BC44BDD0
string: 44444444, value: 1145324612, checksum FFFFFFFF
string: 44444444, value: 1145324612, checksum FFFFFFFF
string: 78007980, value: 2013297024, checksum FFFFFFFF
string: 44444444, value: 1145324612, checksum FFFFFFFF

it looks strange, can you also print $tempNum in that printf ? — catwalk, Nov 24 '09 at 14:29
This code does not generate a checksum. Not even in Perl. :) — bzlm, Nov 24 '09 at 14:32
@bzlm: I am aware of this :) This is just a portion of the checksum operation, the rest of it doesn't seem to be misbehaving so I didn't include it. — dls, Nov 24 '09 at 14:35
Can you please post a short, simple script that actually compiles and runs? Why `for (...)`? What is `print printf(...)?` — Sinan Ünür, Nov 24 '09 at 15:08

score 7 · Accepted Answer · edited Nov 24 '09 at 15:18

7

This was asked at Perl Monks before.

The answer seems to be "use integer".

Please see the original answer at Perl Monks and perldoc integer.

$ perl -we 'use integer; printf("%08X\n",  0xFFFF_FFFF + 0xFFFF_FFFF)'

edited Nov 24 '09 at 15:18

Sinan Ünür

116,958
15
196
339

answered Nov 24 '09 at 14:52

@Sinan: I have to roll back your edit: -M is less wel known than "use", and the link to perldoc for integer is hard to notice - it looks like syntax highlighting. – Nov 24 '09 at 15:17
5

-M will stay less known if you keep hiding it from people. :) – brian d foy Nov 25 '09 at 14:08

Sinan Ünür · Answer 2 · 2009-11-24T15:11:04.003

6

If your perl is compiled with 32-bit integers, integer operations which result in numbers greater than 0xffffffff will cause problems. For example:

my $x = hex '0x1234567890';
my $y = hex '0x1234567890';

print $x + $y,  "\n";

You will get:

Integer overflow in hexadecimal number at ...
Hexadecimal number > 0xffffffff non-portable at ...

Use bignum to add transparent support for larger integers:

#!/usr/bin/perl

use strict; use warnings;
use bignum qw/hex/;

my $x = hex '0x7800798C';
my $chk;

for (1 .. 10) {
    $chk += $x;
    $chk &= 0xFFFFFFFF;
    printf("checksum: %08X\n", $chk);
}

Verify that the output matches you expectations:

checksum: 7800798C
checksum: F000F318
checksum: 68016CA4
checksum: E001E630
checksum: 58025FBC
checksum: D002D948
checksum: 480352D4
checksum: C003CC60
checksum: 380445EC
checksum: B004BF78

Without bignum, I get:

checksum: 7800798C
checksum: F000F318
checksum: FFFFFFFF
checksum: FFFFFFFF
checksum: FFFFFFFF
...

edited Nov 24 '09 at 15:11

answered Nov 24 '09 at 14:28

Sinan Ünür

116,958
15
196
339

Your answer seems a bit cryptic to me. If he keeps adding the numbers and not &= them he will get a floating point number. – Nov 24 '09 at 14:29
if I were to end up with an integer in which the lower 32-bits were set and &= it with 0xFFFFFFFF I would definitely get 0xFFFFFFFF back, but only for that addition. Adding any other number to this summation would result in a unique value. In other words, I will never end up with an indefinitely 0xFFFFFFFF in this summation. – dls Nov 24 '09 at 14:51
4

But you don't end up with an integer. On a 32-bit system the addition results in a float (NV), and then when you try to bitand it with an integer, Perl doesn't know what to do and decides that the integer result will be 0xFFFFFFFF. It's not a perfect choice, but it's a choice. `use integer` solves the problem by denying the promotion to float, so integer + integer gives integer (wrapping when necessary). – hobbs Nov 24 '09 at 18:06

score 3 · Answer 3 · 2009-11-24T14:34:10.410

What result do you want? The &= will delete any bits of the number larger than 0xffffffff. Continually adding numbers together just gets you a larger and larger number, which is then &ed with 0xffffffff. At some point I would expect you would get something other than 0xffffffff, no? But it can never be bigger!

Perhaps a better way to make a checksum would be to xor your numbers together.

my $temp;
my $checksum;

for (...)
{
    #assume $temp is loaded with a new hex string here
    my $tempNum = hex ($temp);
    $checksum ^= $tempNum;
    print printf("checksum: %08X",$checksum);
}

This will make something unique to those numbers.

score 3 · Answer 4 · edited May 23 '17 at 10:32

3

To expand on Sinan's answer, both &= operator and and the %X format are affected by the size of integers compiled into Perl. In this case, the maximum size of either is 4294967295, or 0xFFFFFFFF.

While your variables can hold values larger than this, they will be truncated to this maximum when passed through &= or sprintf("%X").

edited May 23 '17 at 10:32

Community

1
1

answered Nov 24 '09 at 14:34

Adam Bellaire

108,003
19
148
163

ok, this makes sense. Assume that I want to perform all of these additions and then print the 32 least significant bits into a new variable - how would I achieve that? I understand that I can't use &= or sprintf as I originally intended. – dls Nov 24 '09 at 14:40

How can I sum large hexadecimal values in Perl?

4 Answers4

Linked