You've fallen into the trap of trying to use the result to test whether there was an overflow. You can't in fact do that. Signed integer overflow has undefined behaviour, so once it has happened, you're already screwed. You must check for overlow using the operands, before the operation.
Let max
be the maximum representable integer. Assuming the operands are positive, a * b
overflows if and only if a * b > max
. But you cannot perform that test because if a * b
overflows, the result is unusable. Besides, we know that no integer is bigger than max
, so that test would always be false.
So, how can we use the equation, without using the result of a * b
? We shall use the magic of maths and we end up with an equivalent equation: a > max / b
. We only use the integer division operator, which does not overflow, Nice! Now, the equivalence of the equations only holds when b != 0
. Trying to divide max / 0
would be an error. But, we know that a * 0
does not overflow for any a
, so we can trivially implement that special case.
So, what we have is:
int a, b;
// don't forget to initialize
if(b && a > INT_MAX / b)
// overflow, abort!
else
// no overflow, proceed
This only works correctly for positive inputs. I'll leave it as an exercise to the reader to implement the test for all integers.