Infinite loop when using cmpq and je

Question

I'm decrementing RAX on each iteration. If RAX is zero, the program should change flow.

# AT&T syntax
start_calc_factorial:
  decq %rax
  cmpq $0, %rax
  je quit_calc_factorial
  mulq %rcx
  jmp start_calc_factorial

However, the program never terminates. The debugger tells me that RAX has a value of 0xa0257c7238581842 (it probably underflowed, but it shouldn't because of the je instruction). The initial value of RAX is 7.

What could be the problem?

score 6 · Accepted Answer · answered Dec 01 '11 at 18:06

6

I believe the result of the multiplication is stored in the RAX register, so that would definitely mess up the looping.

answered Dec 01 '11 at 18:06

Mark Wilkins

40,729
5
57
110

score 3 · Answer 2 · answered Dec 01 '11 at 18:10

The problem is that you use the same register, rax, as both, the argument and product. Your code is equivalent to this C code:

while (1)
{
  rax = rax - 1;
  if (rax == 0) break;
  rax = rax * rcx;
}

It can loop for a long time if not forever.

What you probably want is this:

while (1)
{
  rcx = rcx - 1;
  if (rcx == 0) break;
  rax = rax * rcx;
}

Infinite loop when using cmpq and je

2 Answers2