Stack Overflow
Stack Overflow

Questions tagged [decimal]

Decimal is the name for our common base-ten numeral system. It may also refer to non-integer values expressed with a decimal point.

Decimal is the base-10 positional numeral system. It is the ordinary numeral system we use; it can be used to represent both integers and (using a decimal point) non-integer values.

Related tags

7532 questions
2
votes
1 answer

Limiting Numeric Digits in Python

I want to put numerics and strings into the same numpy array. However, I very rarely (difficult to replicate, but sometimes) run into an error where the numeric to string conversion results in a value that cannot back-translate into a decimal (ie, I…
mike
  • 22,931
  • 31
  • 77
  • 100
2
votes
3 answers

Always rounding decimals up to specified precision

I'm looking for an elegant way to rounding up decimal numbers always up. Seems like round(0.0045001, 5, PHP_ROUND_HALF_UP); is not returning what I expected so I came up with following function; function roundUp($number, $precision) { $rounded =…
Revenant
  • 2,942
  • 7
  • 30
  • 52
2
votes
5 answers

Java Remove all cases of .0

I want a filter to run through a string and eliminate all of the un-needed .0's from the doubles. I have already tried replaces, but cases like 8.02 turn into 82. I have a feeling this has been done before, but I cannot find any help. I am looking…
Dylan Wheeler
  • 6,928
  • 14
  • 56
  • 80
2
votes
1 answer

MVC 2 - Decimal value is not saved

I'm currently creating forms for contracts, one of the fields is for a rebate %, and has been declared a Decimal in Entity Framework. I'm getting an issue when I enter a value there and save it. Quick overview of the code: EF: tblContract -> Rebate…
Flater
  • 12,908
  • 4
  • 39
  • 62
2
votes
4 answers

SQL Server Float data type calculation vs decimal

In the following query declare @a float(23) declare @b float(23) declare @c float(53) set @a = 123456789012.1234 set @b = 1234567.12345678 set @c = @a * @b select @c select LTRIM(STR((@c),32,12)) declare @x decimal(16,4) declare @y…
Rakesh Singh
  • 49
  • 1
  • 2
  • 8
2
votes
1 answer

Drupal 6 Decimal and Views

Have you ever tried to store a number like this 113044638720122 and then use views to filter it? I was trying to use these large numbers, I used CCK decimal precision to 32 and Views. In a simple test, I passed 113044638720122 as fixed argument,…
Beto Aveiga
  • 3,466
  • 4
  • 27
  • 39
2
votes
2 answers

Deep zoom on a function such as Weierstrass functions - working example or library or starter code for very-high-precision floating point math?

This link shows information, including a "zoom into" the detail of a subsection of a Weierstrass function. It stops, and the notes say that the limitations of the software used to analyze the values for f(x) are hitting the limits of (I'm guessing)…
Warren P
  • 65,725
  • 40
  • 181
  • 316
2
votes
3 answers

Efficient way to convert a hex representation of a number to a decimal value

I have a byte[5] array which represents a decimal number when printed as a hex string. Two digits can be stored in one byte, the hex characters are not used. E.g. [0x11,0x45,0x34,0x31,0x21] -> 1145343121. Is there a more efficient way in Java (some…
user462982
  • 1,635
  • 1
  • 16
  • 26
2
votes
2 answers

Converting masked currency string using Decimal.TryParse

I'm having issue converting this string to a decimal. I tried to follow the documentation here with no luck: Decimal.TryParse Method string stringVal = "-(3434343434.00)"; NumberStyles style = NumberStyles.AllowDecimalPoint |…
mservidio
  • 12,817
  • 9
  • 58
  • 84
2
votes
3 answers

Scientifc Notation Java

I am working on a problem which was given to me by my friend. I need to take the input number in the form x.yzw*10^p where p is non zero and x.yzw can be zeros. I have made the program but the problem is that when we have numbers such as 0.098,…
Jeel Shah
  • 3,274
  • 17
  • 47
  • 68
2
votes
4 answers

The binary equivalent of the decimal number 104

Ok,so I know that the binary equivalent of 104 is 1101000. 10=1010 4=0100 so , 104=1101000 (how to get this??how these both mix together and get this binary?) And from the example here... the octets from "hellohello" are E8 32 9B FD 46 97 D9 EC…
Irene Ling
  • 1,881
  • 3
  • 26
  • 41
2
votes
3 answers

In objective c from a float number I want to sepatate the decimals to do further math work on

I am setting up a calculator that will generate a float number after the user's input. As an example: if the answer generated is 1.75 that would actually represents 1.75 hours. I want to split the answer into two parts to read 1 hr. 45 min which…
Rob
  • 51
  • 1
  • 5
2
votes
3 answers

Java textfield decimal validation

How can I validate user input contain only 5 digits with 2 decimal. I am using below code to check 5 digits, but how to check 2 decimal. if (!id.equals("")) { try { Integer.parseInt(id); if (id.length() <= 5) { …
user236501
  • 8,538
  • 24
  • 85
  • 119
2
votes
2 answers

Storing small decimals MySQL

I need to store a large volume of small decimals numbers (3 digits before the decimal, 6 digits after the decimal). From my understanding of the spec, this will require 8 bytes. I could store the number as an int which requires only 4 bytes and…
DD.
  • 21,498
  • 52
  • 157
  • 246
2
votes
4 answers

LINQ: How to round decimal value for query result?

Possible Duplicate: c# - How do I round a decimal value to 2 decimal places (for output on a page) I have an XML file with decimal values of temperature in string format. Examples: 30 40.6 I retrieve the temperature…
user776676
  • 4,265
  • 13
  • 58
  • 77
1 2 3
…
99 100