1

Problem

as title mentioned, when use math.floor() on 21000*0.7 and 14700.0 the returned value is different.

math.floor(21000*0.7) returns 14699, but math.floor(14700.0) returns 14700.

Lua versions

5.3.x and 5.4.x

Ace.Yin
  • 845
  • 2
  • 9
  • 19

1 Answers1

3

Reason

The behavior you're getting is because of floating point inaccuracies. See this question for more details, but in a nutshell, the reason is similar to why we can't represent 1 / 3 (0.33 repeating) with a finite amount of digits in decimal.

You can test it yourself! Use formatting to get a more precise string representation, and print it:

print(("%.14f"):format(21000*0.7))

Prints 14699.99999999999818!

Lua tostring doesn't have this precision. Some dialects change that because like in your case, it can be confusing. This is also why 0.1 + 0.2 == 0.3 evaluates to false, but printing both expressions individually prints 0.3.

Solution

The solution is to do it this way: 21000 * 7 / 10, assuming that 21000 will be replaced by a variable. If the entire fraction will be unknown, and if 21000 will be replaced by an integer, then you can try doing this: 21000 * 0.7 + 0.1 (as the argument of floor)

Random
  • 505
  • 2
  • 17