Stack Overflow
Stack Overflow

Questions tagged [rounding]

Rounding a numerical value means replacing it by another value that is approximately equal but has a shorter, simpler, or more explicit representation.

Rounding a numerical value means replacing it by another value that is approximately equal but has a shorter, simpler, or more explicit representation; for example, replacing £23.4476 with £23.45, or the fraction 312/937 with 1/3, or the expression √2 with 1.414.

5484 questions
3
votes
1 answer

python 2.7 string format to decimal as round function

I got a problem in string format function which I don't understand. Please help to explain why and how to fix this. thanks. ( python 2.7.3 , [GCC 4.6.3] on linux2 , ubuntu 12.04 x86 ) >>> import locale >>> locale.format("%0.{0}f".format(2), 1.135,…
Hardy
  • 1,499
  • 2
  • 26
  • 39
3
votes
4 answers

Rounding floats with non-exact representation

We have a problem with rounding of floating point numbers for some financial calculations. Basically, we want to round monetary amounts like 1000000000.555 to 2 decimals. However, the float representation of this number is 1000000000.5549999 and as…
Rickard
3
votes
1 answer

MySQL round(x,1) returns 0.0 and -0.0

Tell me if the two zeros make sense: select x, count(1) from (select round(rand()-0.5,1) x from tab) t group by x; +------+----------+ | x | count(1) | +------+----------+ | -0.5 | 1830 | | -0.4 | 3726 | | -0.3 | 3753 | | -0.2 | …
iggy
  • 662
  • 6
  • 14
3
votes
2 answers

Multiplying float and integer cast from string in php by hand gives different results

I resolved the issue with the code that i was working, but this is something that bugs me out. I know that floating point in certain circumnstances just messes up with rounding. I know that php uses types inferences, and there is an automatic cast…
alekhin0w
  • 118
  • 2
  • 6
3
votes
2 answers

Round Excel Time Difference to Next 15 Minute Interval

I have a start time and an End Time in Excel: Start Time 9:15 PM End Time 9:30 PM Time Spent 0:15 I'm looking to round up to the next 15 minute increment, and are using the formula: =ROUNDUP(A1*96,0)/96 However, this rounds up the example data above…
Ryan Ellis
  • 440
  • 2
  • 5
  • 18
3
votes
1 answer

DATEDIFF with minute does not return expected value

Say, I have the following SQL Server 2008 table with data: CREATE TABLE tbl (dtIn DATETIME2, dtOut DATETIME2) INSERT tbl VALUES ('9/10/2012 5:14:10 AM', '9/10/2012 5:15:09 AM'), ('9/10/2012 5:16:12 AM', '9/10/2012 5:18:12 AM'), ('9/10/2012 5:18:43…
c00000fd
  • 20,994
  • 29
  • 177
  • 400
3
votes
5 answers

What is the most efficient way to round a float value to the nearest integer in java?

I've seen a lot of discussion on SO related to rounding float values, but no solid Q&A considering the efficiency aspect. So here it is: What is the most efficient (but correct) way to round a float value to the nearest integer? (int) (mFloat +…
robguinness
  • 16,266
  • 14
  • 55
  • 65
3
votes
9 answers

Rounding errors in python

why the order of multiplications can impact the results? Consider the following code a=47.215419672114173 b=-0.45000000000000007 c=-0.91006620964286644 result1=a*b*c temp=b*c result2=a*temp result1==result2 We all know that result1 should be equal…
Mannaggia
  • 4,559
  • 12
  • 34
  • 47
3
votes
2 answers

Weird rounding in C

Possible Duplicate: C programming division in the following code example the result should yield 130.0 yet once compiled, I get 129.0. Is the right hand side not producing enough precision to get an accurate result? Is there a way around…
illumi
  • 274
  • 5
  • 17
3
votes
2 answers

SQL round function error for value 8.95

In SQL Server 2008: declare @Value float declare @result float set @Value=7.95 select @result=round(@Value,1) print @result /*Prints 8*/ set @Value=8.95 select @result=round(@Value,1) print @result /*Prints 8.9*/ The obtained result is 8.9 for…
Anoop Mohan
  • 329
  • 1
  • 2
  • 13
3
votes
2 answers

How to compute the major unit of a numeric axis

I want to display three labels on the y-axis: min, middle and max. max - min = 9572. The easiest way to find middle is 9572 / 2 = 4786. These values are not nice so Excel would probably increase the interval to 1000 and set middle to 500. How do I…
hidarikani
  • 1,121
  • 1
  • 11
  • 25
3
votes
1 answer

Force a calculation result if less than zero to equal zero in VB.net

Is there a built in VB function to ensure the following: Dim price Dim subsidy if price - subsidy <= 0 then price = 0 end if In practical terms, I've lots of other things going on to calculate price, so I want to simplify this to: Dim price =…
Jamie Hartnoll
  • 7,231
  • 13
  • 58
  • 97
3
votes
4 answers

javascript, issue with number rounding

I knew javascript could have rounding issue with divisions, but not with multiplication. How do you solve those? var p = $('input[name="productsUS"]').val().replace(",", "."); var t = $('input[name="productsWorld"]').val().replace(",", "."); if (p…
Kraz
  • 6,910
  • 6
  • 42
  • 70
2
votes
2 answers

Culture-Based String Formatting For Decimal

I've got a decimal value, which is stored in my database (SQL Server 2008) with a precision of 13, and a scale of 10. Examples: 10.6157894734 68.0750000000 96.8723684210 Basically, the numbers represent a "score" out of 100. Now, i want to…
RPM1984
  • 72,246
  • 58
  • 225
  • 350
2
votes
4 answers

How to round a double to two decimal places in Java?

I am currently try to retrieve a latitude and longitude value from a location. When i convert the location to integer values using the following code: LocationManager locMan; Location location; String towers; private static double…
devinefergal
  • 287
  • 1
  • 4
  • 10
1 2 3
…
99 100